Peer Review - A Paradigm in Peril

The story of two peer-reviewed papers:

In January 2009, Nature splashed its front cover with the results of a new study titled 'Warming of the Antarctic ice-sheet surface since the 1957 International Geophysical Year'.

The article was accompanied by a glowing editorial from Nature and was widely reported on in the media. A very short time after the paper was published, a number of factual errors were found in the paper, along with significant issues with the methodology used to obtain the surprising results. The errors and the methodological problems were reported and discussed by climate change blogs Watts Up With That, The Air Vent, Climate Audit and Real Climate.

Imagine if at this stage Nature's editor in chief looked at the reported blog commentary and decided the journal had published a paper, which while it had gone through the normal peer review processes, based on some of the blog commentary, was basically fundamentally flawed and should not have been published.

Furthermore, the original reviewers may have shared some of the climate alarm notions of the authors, bringing the veracity of the original review into question. Media coverage also sensationalised aspects of the results. The editor in chief is so embarrassed by the publication of the erroneous paper, he decides to resign. as a peer reviewed paper, by Ryan O'Donnell, Nicolas Lewis, Steve McIntyre and Jeff Condon, in the

Sounds farcical? In fact Nature's editor did not resign. Indeed there was no need to resign, there was no expectation on the part of the scientific community that a resignation was called for, regardless of the issues with the paper.

Subsequently Nature published a correction by the authors that dealt with some of the factual errors. And later, the blog commentary dealing with the methodological problems, ended up being published in the Journal of Climate.

Unlike the original paper however, this received very little media attention. Perhaps the long time the paper spent in peer review (10 months) and the less sensational results dulled the media's interest.

And the second paper?

To some astonishment the scenario outlined above, in which a journal editor resigns over the publication of a controversial paper, has recently occurred. It involves a paper by Roy Spencer and William Braswell published just last month in the journal, Remote Sensing titled 'On the Misdiagnosis of Climate Feedbacks from Variations in Earth's Radiant Energy Balance'. Like the Antarctic paper in Nature, the paper by Spencer and Braswell went through the normal peer review process. It was promoted by the authors' university through a press release and received a few mentions in the media.

Like the Antarctic paper, some of the media coverage sensationalised the results. The paper also came in for favourable and harsh criticism on the internet, and it appears the paper is not free from error, or methodological issues.

However, rather than allow the peer reviewed system to take care of the issues in the normal manner, the journal's editor, Wolfgang Wagner, took the unprecedented step of resigning over it. In his editorial comment in Remote Sensing, Professor Wagner explains how he, remarkably, relied not on the peer reviewed literature to back his decision, but on comments on an internet blog. He states:

Peer-reviewed journals are a pillar of modern science. Their aim is to achieve highest scientific standards by carrying out a rigorous peer review that is, as a minimum requirement, supposed to be able to identify fundamental methodological errors or false claims. Unfortunately, as many climate researchers and engaged observers of the climate change debate pointed out in various internet discussion fora, the paper by Spencer and Braswell that was recently published in Remote Sensing is most likely problematic in both aspects and should therefore not have been published. After having become aware of the situation, and studying the various pro and contra arguments, I agree with the critics of the paper. Therefore, I would like to take the responsibility for this editorial decision and, as a result, step down as Editor-in-Chief of the journal Remote Sensing.

In the process the editor has also broke the trust of the reviewers, deriding them in the process by stating in his editorial:

The editorial team unintentionally selected three reviewers who probably share some climate sceptic notions of the authors.

God forbid that a paper ever be reviewed by scientists who may have similar opinions to the author. On this basis will we now see "sceptical" reviewers invited to provide critical input into the next IPCC report?

Professor Wagner, who has no expertise in the relevant area of climate science and hence cannot judge the value of the paper on his own the merits (that's why expert reviewers are used), relied on the commentary of a non-peer reviewed climate blog in order to justify his resignation. The resignation was not expected, or required and is highly unusual. The paper by Spencer and Braswell has not been retracted, however a number of formal critiques of the paper have very rapidly appeared. One in the journal Geophysical Research Letters and one in Remote Sensing. Commentary about both has also been quick on the internet (eg, at Climate Audit - here and here), but no resignations are expected from the issues uncovered.

The commentary in Remote Sensing was received and accepted at warp speed, all in the space of 24 hours; and oddly, against convention, it does not appear with a reply by the original authors. It is anticipated that this will be published in the near future.

In comparison, criticism of Nature's Antarctic warming paper sat in peer review at the Journal of Climate for an amazing 10 months. It seems an expressway exists through the peer review system for papers and comments that support a particular view, while others are considered and reviewed at a snail's pace. Nevertheless ,the system works in slowly advancing knowledge, even if sometimes we go back a few steps in order to progress further in the future.

Peer review is a dying paradigm.  It made sense in an era when journals were strictly dead-tree, slowly produced, slowly edited and extremely expensive source of research material by and for scholars.  In an era when a paper can be written, edited, graphs created, and the final product made available to the world on free platforms (heck, Google blogger is probably a permanent as the average research library) with the push of a button, there's simply no excuse for having such a laborious and error prone mechanism.  If you're going to make errors anyway, you might as well make them fast.

At it's best, peer review is a way to keep the volume of scholarly literature down to an acceptable minimum. I would say it has been largely unsuccessful at that mission; scholars have a strong motivation to publish, the phrase "Publish or Perish" is not an idle comment.While quality is desirable, quantity is a must, so scholars are adept at getting papers of minimal value through the system to publishers who produce the product for a profit (not that there's anything wrong with that).  At it worst, peer review is a system that can be manipulated by the scholars to help their friends and colleagues.  You don't even have to deny a paper publication; just slow it down long enough that your own student's thesis on the topic can be published first, and the thunder is largely stolen.  Despite belief that repeatability is a hallmark of solid science, getting there second or third doesn't win any prizes.

I have come to the conclusion that peer review is like the proverbial Vietnamese village, it must be destroyed to be saved.  Let papers, or more like electronic manuscripts, be published willy-nilly.  Let the authors publicize it by whatever means they deem appropriate, and let commentary and criticism flow freely.  In such a system I would expect most papers to do what most papers do now; sink like a stone and leave few ripples behind.  But the important and controversial ones, like the ones discussed above would still be read, discussed and evaluated.  And that's what counts.