Economist, Fix Thy Own Field

The usually sensible Megan McCardle, an economist has some thoughts on fixing science?

Research That's Just Too Good to Be Tested

We know what happens when science is "politicized": Think of global warming. Politicizing science leads both sides to retreat into bunkers, hurling insults at each other, and trying to cut each other off at the knees by any means necessary.

But what happens when science isn’t politicized? Part of the answer may be: the epidemic of replication failures we now seem to be seeing. A recent paper from the Fed argues that economics has problems similar to those recently found in psychology: A lot of research results are getting published, and a lot of the interesting findings can’t be replicated, often because key data or instructions aren’t available.

Now, that is not, by itself, necessarily a problem. As I’ve written before, “finding an interesting result that fails replication” is an important part of science. We should not expect every paper to get a replicable result, not even papers that are meticulously done to the highest research standards. The outliers, the coding errors, the unforeseen model weaknesses -- these we will always have with us.

But “the authors did not provide enough data to replicate their work” is not a problem science should ever have; neither is “a weak result lived on in the literature for years before anyone tried to repeat it.” I read these papers about replication failure and think “Aren’t scientists supposed to be competitive? Why aren’t these guys trying to destroy each other? Or at least provide a reality check? How has this gone on for so long? Why do so many journals allow authors to publish without providing the necessary tools to replicate their work?” . . .

The problem with replication is that it is. well, replication. There are really strong incentives in science to do novel science. It's one of the first criteria for consideration in the review and publication of any paper. And completing and publishing papers is the touchstone of scientific careers, both in tenure and promotion decision, and in funding, both of which are necessary for a long, successful run. And you don't get funded for  replicating someone else's work unless it is controversial and you won't publish a paper that essentially says "Jones was right" unless there was some substantial worry that Jones was wrong.

One of my very first scientific papers was a paper that refuted a paper by another well known worker in the field. Something that he thought was a biological effect, turned out to be a mere chemical artifact. Not a big deal, but a cautionary tale. The work was unfunded, really just project being carried out on the side by a professor and myself.

In turn, years later, a student of a grad school colleague overturned one of my early papers, on a similar issue. What comes around, goes around.

If the problem of lack of replication is so bad (and it is, particularly in medicine, where people die waiting for advances), government, who provides most of the money, needs to change the incentives, by specifically providing funds for studies aimed at identification of suspicious studies, and their replication. This would be a good job for grants to grad students and post docs.