Start With The Tycho Brahes

I was reading an interesting editorial in the May issue of Analog magazine (unfortunately not online) about the scientific method, by editor Stanley Schmidt.

Schmidt, who holds a PhD in physics, pointed out that there are a lot of scientific fields where the classic experimental method is untenable, for example astronomy.

But what about astronomy, which practically everybody considers clearly one of the most basic and important of sciences? Who has ever set up an experimental pregalactic gas cloud to watch how stars and planets form? Or even a protostar in a laboratory to test a theory of stellar evolution?

So astronomers and astrophysicists, in general, can't do controlled experiments. the have to make do with observing what nature provides--and they can't even get close to that. As astronomer and SF writer Michael Brotherton succinctly put it at Launchpad[...], "Everything we know about stars we know by analyzing light [in the broad sense of 'electromagnetic radiation,' of any wavelength] from very far away."

And this is what struck me the most:

But if the earliest students of what we now call physics and chemistry had disdained to study them because they weren't understood with twenty-first-century rigor, they would never have gotten off the ground. No science starts off at that level; somebody has to do the groundwork before rigorous theories can be formulated, tested, and refined. As my brother Dennis (a computer professional with a strong interest in psychology) once said, "I used to think psychology needed a Newton. Now I realize it isn't ready for a Newton. First it needs a Tycho Brahe"

By the way, Tycho Brahe is said to have died from from drinking too much wine and not being able to relieve himself due to court etiquette (although recent evidence suggests it might have been simple mercury poisoning), Brahe certainly wasn't the last person to drink too much in Prague. I think the Tycho Brahe analogy is especially apt when it comes to nutrition research. Brahe wasn't a theorist but a collector of data, greatly ahead of his time in astronomical accuracy. Brahe was the equivalent of a modern experimental astrophysicist with Kepler being the theoretician. 

When I see nutritional studies quoted that use self-reported data, or don't control for people who dropped out of the study or other idiotic things that are supposed to pass for science I have to shake my head in wonder. What's so annoying is to see the language of science used in these papers, statistical regressions, big words, blah blah blah, when the raw data is nonsense.

What many of the 'soft' sciences need more than anything else is good data. Maybe a better way to put it, what many of the complicated sciences need is better data, better controlling for variables. 200 similar studies with 60 participants are not nearly as effective as one study with 1200 participants. Let's say you divide the participants into three groups, intervention A (say low-carb), intervention B (low-fat) and a control group (no intervention). All these small similar studies with 20 people per group are going to vary somewhat in implementation so they will be less effective than a single large study. Rigorously measuring 1200 people in a dietary study is, of course, very expensive. But no more expensive than all the small studies that are now done that are essentially statistical noise.

Instead we are left with meta-studies, AKA the cherry-pickers utopia. Okay, not all meta-studies suck, but even the most conscientious meta-study is simply not going to be able to control for all the study variances and hence is going to skew the data.

The real problem is that accurate data is extremely difficult to collect in complicated fields like nutritional or health studies (or economics or psychology). Just as there's no crying in baseball, there's no placebo effect in physics. Or maybe there is? Perhaps that's the secret of the Theory of Everything! I'm off to write a grant proposal.

This reminds me of a recent criticism I made of archaeology. I wrote:

So a problem I see with the paper from Miki Ben-Dor who has a blog entitled Paleo Style, or with archeology in general it is pretty easy to cherry-pick or insert one's biases, because it's all in retrospect and the data is so scant.

What I was really thinking is that archaeology is observational hence not as rigorous as experimental science. But the reality is that archaeology is much like astrophysics. It is about uncovering data and analyzing that data based on increasingly rigorous methods and frameworks (such as genetic analysis, population modeling, etc). So, my bad.

No one has ever seen a black hole, but we have plenty of indirect evidence they exist just from observing photons that reach us from distant parts of the universe. Likewise, these same observations have led astrophysicists to postulate dark matter and dark energy, opening up some of the most interesting new topics in the world of physics today.