Not All You Read is True

The Truth Wears Off

A truly great article on the tendency for science to come forth with "New Findings" that fade upon closer examination. The article is long, but filled with interesting, and important cases where science initially produced surprising results, which either turned out to be much weaker than originally observed, or disappeared all together. Potential contributing factors for the effect include:

• Early positive statistical flukes are eventually overwhelmed by the mean of no effect.
• Subconscious bias by the investigator in selecting subjects or making measurements
• Publication bias: If only papers with positive results get published (largely but not entirely true), and 5% of statistical tests give a false positive, the population of papers will contain an inordinate number of false positive results.

Just out of Grad School, I published a paper with a professor of mind on a mildly obscure (OK, really obscure) process in the nutrition of diatoms.  We used an unusual diatom to show that the uptake of Si was a function of the concentration of a particular form of Si, mediated by pH.  After I left school, and got a sort of real job, we tried to get funding to prove it was a general phenomenon.  But, having no success at that, we gave it up, and dreamed of it wistfully.

Many years later, a student of another student of the professor (grandstudent?), succeeded in proving that we were wrong, it was not a general rule, and probably just a statistical fluke (see #1 and a little #2).  I had the pleasure of congratulating her at a meeting.

So when you read about all the nifty science I post, remember, a lot of it is probably wrong, and some of it may never be checked again.