A truly great article on the tendency for science to come forth with "New Findings" that fade upon closer examination. The article is long, but filled with interesting, and important cases where science initially produced surprising results, which either turned out to be much weaker than originally observed, or disappeared all together. Potential contributing factors for the effect include:
- Early positive statistical flukes are eventually overwhelmed by the mean of no effect.
- Subconscious bias by the investigator in selecting subjects or making measurements
- Publication bias: If only papers with positive results get published (largely but not entirely true), and 5% of statistical tests give a false positive, the population of papers will contain an inordinate number of false positive results.
Many years later, a student of another student of the professor (grandstudent?), succeeded in proving that we were wrong, it was not a general rule, and probably just a statistical fluke (see #1 and a little #2). I had the pleasure of congratulating her at a meeting.
So when you read about all the nifty science I post, remember, a lot of it is probably wrong, and some of it may never be checked again.