Part 6: Peer review
The discussion of peer review is the most replete with basic misconceptions about the scientific process. Pearce appears to conflate any rejection of a paper or even a negative review for any reason as a prima facie case of mainstream climate scientists … censoring their critics. But in none of the cases highlighted were anyone’s view ‘censored’. To have your opinion published in peer-reviewed literature is not some fundamental right – it is a privilege that depends on your ability to do the analysis and the marshal the logical arguments and data to support your point.
Pearce, surprisingly for someone who has been on a science beat for a long time, states that peer review is the supposed gold standard of scientific merit. This is not the case at all. As we’ve outlined in many articles, peer review is just a first (necessary) step towards scientific acceptance and as the number of badly flawed papers that do appear in the literature attest, it is no guarantee of merit. For it to work of course there need to be some standards that should ideally be met, and this will lead to the rejection of some submissions. Thus automatically equating rejections of bad submissions with squashing of ‘dissent’ is like assuming that anyone who gets an F on a test is being unfairly discriminated against.
Pearce also declares that the mere act of reviewing a paper that is critical of your own work is mired in ‘conflicts of interest that would not be allowed in most professions‘. This is wrong on multiple levels. First of all, peer review of the literature is hardly unique to climate science, and so his claim about improper conflicts of interest is an accusation against the whole of science, not just climatology. Secondly, he confuses the role of the reviewer with that of the editor. Editors often solicit reviews of a critical comment directly from those being criticised, since that is often the easiest way to judge whether the critique is substantive. That is not the same as giving the right of veto to the criticised authors since, of course, it’s the editor’s job to weigh the different reviews from different sources, and use their own judgment as to the merits of the critique. Not asking the original authors for comment can certainly be (and has been) problematic and unfair to them. The problems most often arise – such as in Soon and Baliunas (2003) or McIntyre and McKitrick (2003;2005) when the criticised authors are not involved at all.
In the cases mentioned in this article, there is absolutely no evidence of unfair discrimination. Indeed, in one case of a submission by Lars Kamel, the reasons for rejection are obvious and Pearce appears not to know what the criteria for acceptance even are. He states that “the finding sounded important, but his paper was rejected by Geophysical Research Letters (GRL) that year“. But papers are not accepted or rejected because a finding ’sounds important’, but because that finding is backed up by analysis and logic while acknowledging the prior work on the topic. In this case, the author did not “however, justify that conclusion with any data or analysis“, and so a rejected manuscript would have been very likely, regardless of who the reviewers were. Similarly, the assumption that “some would have recommended publication” purely because it called into question previous work is unsupportable as a general rule. Filling the literature with papers ‘just asking questions’ that ’sound important’ but not demonstrating any actual results is a recipe for wasting everyone’s time with poorly thought out, and even mendacious, critiques of mainstream science from HIV-denial to perpetual motion machines. Papers in the technical literature are not just opinion.
Pearce also assumes (without evidence) that Kamel was discriminated against because Jones “would certainly have been aware of Kamel’s [negative] views about mainstream climate research“. But why should this be assumed? Most scientists (luckily) go through their whole career without wasting their time investigating and cataloguing the cranks in their field. Some climate sceptics get addressed here on RC a fair bit, but it would be a big mistake to think that these people, particularly the more obscure ones, are the subject of water cooler conversations at climate research labs across the world. Indeed, I can find no reference to Kamel on RC at all and I was unaware of his peculiar views until this story emerged. Why Jones should be assumed to omniscient on this topic is unclear.
Pearce quotes McIntyre discussing “CRU’s policies of obstructing critical articles in the peer-reviewed literature” slowing the resolution of unspecified “issues”. This is simply disingenuous – what papers have been obstructed that would have resolved what issues? We are unaware of any such papers, and certainly none from McIntyre. Prior therefore to declaring that “evidence, flawed though it might be, is actively being kept out of the journals” it behoves Pearce to actually find such evidence. Otherwise, the simple non-appearance of these mythical critiques is apparently proof of the corruption of the peer review process.
As an additional example of problematic practice, Pearce highlights a June 2003 email from Keith Briffa, who as an editor ‘emailed fellow tree-ring researcher Edward Cook, a researcher at Lamont-Doherty Earth Observatory in New York, saying: “Confidentially I now need a hard and if required extensive case for rejecting [an unnamed paper] to support Dave Stahle’s and really as soon as you can.”‘. However, without context this is meaningless. People often sign reviews and this could well have been a second go around on a particular paper whose first round reviews would have been seen by everyone concerned. Briffa (like many editors) can have a feeling that a paper should be rejected for multiple reasons but would like to have the reasons gone into in some detail, mostly for the benefit of the authors. This is one reason why reviewing bad papers is so much more work than good ones. Quoting this as if it absolutely demonstrated bad faith or misconduct is simply a smear.
Pearce then accuses Cook of some unjustified quid-pro-quo because he wanted to use some of Briffa’s data to assess the practical implications of a new analysis technique, that Pearce interprets as “attacking his own tree-ring work“. However, this too is a misreading. The work in question has subsequently been revised and the authors themselves have said that the current submission is improved over the initial submission. It goes along with the overall point made above, that pure criticism is not particularly useful – it is much better to demonstrate that some technical point actually matters. This is what Cook appears to be asking for help to demonstrate.
The article then moves on to the issue of the 2003 Soon and Baliunas paper in Climate Research. Pearce nowhere acknowledges that it is (and was) widely regarded as a complete failure of the peer review system. Six (very independent minded) editors resigned from the journal because of the publisher’s inaction on tightening up peer review standards and even the publisher himself declared that the paper’s conclusions were not supported by the data or analysis of the authors. Is this not germane?
Pearce suggests that the reaction to the demonstrably low standards at Climate Research involved “improper pressure“. This has no validity whatsoever. The suggestion was made that maybe people should not submit work to the journal or cite work that appeared there. But how can a suggestion made among colleagues and not transmitted more widely be ‘pressure’ of any sort? People have their impressions about journals determined by many factors, and if they are seen to be publishing bad papers, that will be noted. Compare the reputations of Science and E&E for instance. Which would you rather be published in if you had a good paper?
The one email that Pearce declares “means what it seems to mean” refers to the declaration (along with exclamation point) that Jones would “redefine peer-review!” rather than include two flawed papers in the AR4 report. But it should be obvious that no-one gets to redefine what ‘peer reviewed’ means, and the exclamation point underlines the fact that this was hyperbole. The two papers referred to (McKitrick and Michaels, 2004; Kalnay and Cai, 2003)) were indeed discussed in Chapter 2 of AR4 as the contributing lead author of that chapter Trenberth rightly pointed out. As an aside neither have stood the test the time.
The problem with lapses in peer review (which will inevitably occur) is that they are sometimes systematic, indicating a more institutional problem instead of simply an unfortunate combination of poor reviewers and a busy editor. This appeared to occur at Geophysical Research Letters over the period 2005-2006. There was a string of bad papers published – ones that did not properly support their conclusions and made basic errors in the science. For instance, Douglass and Knox (2005), Douglass, Patel and Knox (2005), Douglass, Pearson and Singer (2004), Douglass, Pearson, Singer, Knappenberger, and Michaels (2004), and Loáiciga (2006).
Science is indeed a ’self-correcting’ process, but someone has to do that correcting, and scientists do get frustrated when they have to spend weeks dealing with the aftermath of bad papers in the media and putting together the comments that almost every single one of these papers generated. (For amusement and for an example of the lack of standards being talked about, look at the response of Bjornsson et al to the Douglass, Patel and Knox paper).
Are scientists supposed not to notice these patterns? Or never discuss them among colleagues? The implication that the mere discussion of the situation is somehow a corruption of the peer review process is completely unjustified. Peer review only holds the status it does because scientists are on guard against failures in the system and try to correct them when they occur.
Update: Coincidentally, David Adams on the Guardian makes many of the same points as we do.