Is peer review as great as it's cracked up to be?

Great Debates
1

I think that it is not. As I see, the nature of the academic world does not lead to the publication of high-quality, truthful articles. In many cases it leads to the opposite. To understand this, let’s look at the pressures on the writers and the reviewers.

Everyone in academia is under immense pressure to publish. From grad students to tenured professors, publications are important for earning degrees, getting jobs, getting promotions, getting pay raises, receiving grants, and just about everything else. The situation is often described as “publish or perish”. Faced with such a choice, it’s no surprise that most academics would prefer to publish. Hence they may submit for publication work which is flawed or incomplete, not because they want to do so, but because the system all but forces them to do so.

On the review side, the system provides very little reason to do thorough reviews. The reviewer receives few tangible benefits for doing a review. Academic are busy people, so it’s likely that a thankless task like reviewing an article will get a low priority. This may lead them to skip the review entirely or to give the article just a cursory reading, rather than the many, careful readings that would be needed to spot all the flaws. Again, it happens not because the academics want it that way, but because the system makes it that way.

Furthermore, academics are humans, and humans prefer reading interesting things to reading boring things. Let’s face it. Most academic journal articles are as boring as reading the phone directory. This provides yet more reason to skip or skim when reviewing.

This is not an abstract theory; it has been demonstrated in reality. Physicist Allan Sokal once sent a paper on Transgressing the Boundaries: Towards a Transformative Hermeneutics of Quantum Gravity to a leading journal of sociology. The paper was pure nonsense, yet it was published without any revisions. There have been several similar hoaxes over the years.

This sort of thing happens in academic journals, but it could not happen in a mainstream newspaper or magazine. Those sources might print erroneous articles, but they wouldn’t print pure nonsense because they have readers. For many academic journals, the readership is very small. Some articles probably never get read by a single person after publication. Hence they can get away with publishing nonsense. That’s why I tend to view peer-reviewed articles with a healthy dose of skepticism.

(Let me address two responses that have come up when I’ve mentioned these ideas in other threads. The first is that I’m trying to dismiss all academic research. I am not. I am merely questioning the dogma that academic journals are automatically reliable. The second is that I am insulting academics. There again I am doing no such thing. I am, in fact, saying that academics are humans and that they do the logical things giving their circumstances. What I’m criticizing is the system.)

2

I disagree!

3

Social Text is not a leading sociology journal. It is a post modernist bullshitfest which did not peer review its articles at the time when Sokal’s paper was published.

4

FTR: 2.5 is correct.

It’s my understanding that a lot of stuff in Physics (and maybe the other sciences as well) is happening in online databases of articles that are not peer-reviewed. (Someone more in the know than me please either elaborate or disconfirm.) And really, with communications technologies as they are, peer review does seem to me to be less necessary. In the past peer review served to keep people from having to spend time sifting through all the junk to get at the good stuff. But these days, it’s so easy, quick and cheap for me to access any paper anyone has put up online that it is much less of a burden for me to do my own “reviewing.”

It’s always been the case that anyone could tell with a quick skim whether a paper was going to be worth reading more closely. But the problem was there was no easy way to have access to all the stuff that was at least worth skimming. So you had the journal system, and the peer review process, to act as a sort of pipeline and filter to concentrate all the available quality material into an easily accessible form. Nowadays, though, you can have easy access to all the skim-worthy material with, basically, a wave of the hand. And its easy enough to tell at, basically, a glance, which bits are worth reading and which bits aren’t.

But I don’t think there’s anything wrong with peer-review. I don’t think there’s good evidence that it leads to mediocrity. The one example you gave–Social Text–turned out (as subsequent posts have shown) not to be a good example. And my own experience with the so-called “first-tier” journals" is that the stuff in them is pretty nearly always damn good stuff.

-FrL-

5

I’m not an academic, but I work in industry and am very active in conferences and journals, as a writer, program committee member, editorial board member, program chair, and special issue editor. I’ve done at least a hundred reviews, and probably have read several hundred.

First, the reason academics are pressured to publish is that the whole purpose of research is to advance the field, which you can’t do without publication. The purpose of industrial research is to make money, which is why most industry people don’t publish nearly as much as academics.

Those who write junk, and there are some that do, find their papers rejected, their publications records inadequate, and their tenure chances slim. Not a good career move. We in the center of power know the good guys from the bozos. I suspect most fields are similar.

It’s far more interesting if it is in your area of expertise. We have an interest rating, and those in the area (who are best able to review) universally give higher interest scores than those who don’t. I’ve seen very detailed reviews and very superficial reviews. Typically a paper gets reviewed by 3 - 7 people, with the hope that someone will catch mistakes. This is not to say that all mistakes get caught - that isn’t the intention of peer review.

Newspapers and many magazines are on tight deadlines, and so there isn’t time to do the kind of review done for journals. Hence the large number of corrections in the Times every day. There may be few readers of specialized journals, but those who do read a paper are usually more qualified to critique it. Back 30 years ago a real boner got into a highly respected journal, and in two months there were several letters published pointing out the mistake.

In any case, interesting results will be reproduced, if possible, which is yet another check.

A dogma no one who has any connection to them holds. Peer review isn’t perfect, the reason for the request for a peer reviewed cite is that it is a damn sight more reliable than some guy dumping something on a website or even publishing it in a book.

So, how much do you really know about the system? How many papers have you published, how many reviews have you done, and how many conferences or journals have you been responsible for?

6

Putting a draft on the web is great, because many journals have years long queues, so publication is not timely. However, aren’t a lot of these in the queue for “real” publication anyway? My academic friends are very sensitive to the publications that count for tenure committees and those that don’t. Our conference has some tracks that are less reviewed, to get industrial people more involved, and the academics insist that the distinction between solidly reviewed papers and less reviewed papers is very clear.

7

Yes, anyone who read your responses in SentientMeat’s thread from a few months ago knows this. And your (non-)responses got you a Pit thread (that was more than deserved, IMO).

Personally, I see peer-review as analogous to democracy in this sense: it may not be perfect, but it’s the best we’ve got and thus provides a minimum standard. No one who has published a paper under the peer review system would claim that it makes a paper “automatically reliable”, just as no scientist (or philosopher, for that matter) would claim that science can provide irrefutable proof of synthetic knowledge. Scepticism is good. Being able to discern when it is warranted is even better. Contrarianism is bad, as is willful ignorance.

But let’s get to the crux of it. You give four reasons that peer review leads to the opposite of “high-quality,truthful articles”, which I’ll summarize as:[ol]
[li]Publish or perish pressure[/li][li]Shoddy reviewing[/li][li]Human imperfection[/li][li]Audience size[/li][/ol]
What process would you suggest that would mitigate or solve all these issues in one fell swoop (assuming they should be, or even can be, solved)? For instance, how would you address human imperfection without relying on some egotistical notion of possessing divine knowledge? Unless you’re delusional, my contention is that you cannot.

With that said, and recognizing that you have a very particular axe to grind on this subject, can you suggest a better process?

8

Despite your assertions, this paragraph actually contains a strong argument for peer review, not against it: taking for granted that ‘for many academic journals, the readership is very small’, those journals lack the fact-checking apparatus that the general public provides for mainstream newspapers, and thus, were they to abolish peer review as well, could ‘get away’ with publishing even more nonsense (which wouldn’t exactly be in the interest of any journal, by the way).
So your argument in that paragraph is essentially that because journals try to keep the nonsense to a minimum via peer review, you don’t trust peer review, which doesn’t follow, because the abolishment of peer review could only lead to an increase in nonsense.

Besides, the reviewers of a paper will often have a strong interest in the paper’s claims, because they’re working in the same or a related field and may have either different or similar views, and thus their own work stands to be either called into question or supported by the paper. From that, it also follows that the more controversial the claims are, the stronger the interest will be.

9

To add to my experience with the entire system:

  • Peer review is absolutely not designed to detect fraud. If you poke it with a stick, it will collapse. But peer review depends on fraud not happening in the first place. There are other mechanisms to deal with fraud but it’s frankly a hard problem. The best solution seems to be in graduating scientists who aren’t fraudulent.

  • A lot of why peer review works is the sheer idealism of the practitioners. Scientists believe in the enterprise of peer review and so they are willing to dedicate significant amounts of time for a largely thankless task because that’s what’s necessary for science to progress. It doesn’t take many of them to believe this, just enough

  • There are quite a few acknowledged flaws with the peer review process but these are all incredibly hard problems to fix.

  • There is a lot of exciting work going on with other experimental models. Open reviewing, Open publishing, wiki style approaches etc. So far, none of them seem to scale the way peer review does.

  • At least the academic communities I’m most familiar with are all incredibly introspective so the peer review process is an active topic of discussion and everyone would like to find ways to improve it keeping in mind all of the existing constraints of academia.

10

Do you have a better alternative? Who would be better able to judge the value of a paper than somebody who works in the same narrow field and has to be well informed on what is being published by others in the same field? In addition, peer reviewers not only decide whether or not a paper is accepted or rejected, but frequently make acceptation dependent on changes that improve clarity, discuss implications of the result or even the performance of additional experiments to exclude alternative explanations for the results presented in the paper.

I have taken part in this process both as an author and as a reviewer, mostly in journals with a high impact factor and correspondingly tough review criteria. I have often gotten suggestions from reviewers that have helped me to improve my manuscript, and I try to do the same in my reviewing.

The reputation of a scientific journal is highly dependent on the reputation of its reviewing process. When you have an article to be published, you make a decision on where to publish dependent on your own assessment of the quality of the article - you submit it to the best journal that will accept it with a reasonable probability. You submit your best work to the journals you judge to be the best ones in the field, while for the more run-of-mill articles you lower your goals.

11

Interesting OP, albeit containing one or two fatuous comments. Academic journal articles are ‘boring’? Sure…Ask any academic what they most value in their job, and would do 10 times more of if they had the time? ‘Reading the literature’ - Everything stems from it.

Anyhow, one dimension missing from the OP that reinforces peer review is the academic’s self interest in publishing quality, reproducible work. When you submit a scientific paper, you hang your balls out (that’s how you become great, man :wink: ), and there is no shortage of people queuing up to aim a kick at them. Putting an irreproducible paper into the literature would be a major, major problem for any academic - could and has ended careers.

Peer review is actually a much more contentious issue at the other end of the process IMO, reviewing grants. A lot more at stake.

12

Actually, getting a reputation of publishing crap is not very good for your longtime career. Nowadays, it’s not enough to have a long list of publications, both the impact factor of the journals and the number of times your articles get cited are used to for ranking.

One of the questions you have to answer when reviewing a paper is whether or not the results are sufficiently novel and interesting (to people working in the same field, of course) to be published in that journal, and whether it fits the scope of that journal. I am quite aware that a result that is very exciting to someone working in the immediate field may be utterly boring to someone working on different problems.

The reputation of a scientific journal depends on the quality of the review process. You are normally called to review in the journal in which you yourself publish. Thus by allowing crap to be published in the journal, you indirectly devaluate your own publication.

i somehow suspect that it would be much more difficult for a sociologist to pull over a similar hoax in a reputable physics journal.

I 'll resist the temptation of naming certain newspapers frequently found at supermarket checkout counters that can be relied on to publish every crackpot ufo sighting, conspiracy theory, celebrity blooper …

13

But all of this is changing, as you must know. At first, they tried to speed up the process by issuing shorter, timelier “Letters” journals (So Physical Review spawned “Physical Review Letters”, for instance), but with the internet age, we’ve transcemnded that, and hasve peer-reviewed internet journals like Optics Express, which are every bit as respectable as paper journals

http://www.opticsexpress.org/Issue.cfm

14

I hear you re the problems of peer reviewing and academic publishing, but is there really anyone who’s arguing that peer review is in fact a flawless system? Who is doing the cracking here?

15

At least make them anonymous. Remove authorship credits.

16

There’s many journals that actually work that way - is it really also done non-anonymously? (nomynously, I suppose :))

17

My husband has published and reviews papers in the biological field. I don’t think authorship is ever hidden. Same with grant proposals (although perhaps with greater justification, given the value of a track record).

18

:dubious:

This may be the exception, but definitely not the rule, that’s for sure.

19

I think it depends on the field. In physics, the way it is usual done is one-way anonymous, i.e., who the reviewers are is not known by the authors, but the reviewers know who the authors are. I once talked to an editor at Physical Review who said that, technically speaking, authors could request that they be anonymous to the reviewers but that such requests were very rare and, in his experience, were usually made by authors who were basically nuts. It is also often hard to make authorship anonymous when you are citing your own previous work…and, in fact, this editor told me of a case where the author asked to be anonymous despite the fact that one of the first sentences of the paper was, “In my previous work…” complete with the references.

However, my sister is in public health and she told me that in that field the authors are usually anonymous to the reviewers. And, I think I did look at the advice to authors in one of the journals and they actually advised the authors to try to cite their own work in ways that doesn’t give away that it is theirs.

As for the main subject of the OP, first I think you probably need to distinguish what fields one is talking about since I think peer review in the physical sciences is probably pretty different than in the medical field which is probably pretty different in the humanities. I only feel qualified to talk about the physical sciences.

In the physical sciences, I think peer review is an imperfect, but still important, filter. It certainly doesn’t guarantee that the papers published aren’t garbage but I imagine it does eliminate a lot of the garbage and also force authors to be clearer, less polemical, and do a more thorough citation of the previous literature. I’ll admit that when I look at many papers that do get published, even in the prestigious Letters journals, I do sometimes wonder how they got through. But, I imagine things would be much worse without it.

And, looking in the field of climate science for example, there is so much garbage out there on the web and way, way less if you stick to the reasonably prestigious peer-reviewed journals. Plus, “skeptic” scientists like Lindzen and Spencer, who will often say some quite crazy stuff in op-ed pieces and the like, don’t seem to be able to get away with that in the peer-reviewed journals. So, what they publish there, while still quite possibly (likely?) wrong, has to at least be less polemical and the claims they make much more measured.

20

My wife publishes in and reviews for biomedical journals. Her field is small. Thus when she gets a review back she can make an educated guess about who reviewed it based on the objections. Never the less the journals try to keep the reviewers anonymous-but never the authors.

My wife reviews a lot of stuff, for a reason. Firstly she interacts with the editors at journals where she is likely to submit this doesn’t help her reviews but it has helped once accepted. Secondly she is reading research which is relevant to her area-before other people do.

Personally I think that the biggest problem with scientific journal publication is that there is a bias toward positive results. I am not sure if peer review contributes to that or not.