# Is Peer-Review Valid?

**URL:** <https://boards.straightdope.com/t/is-peer-review-valid/682928>\
**Category:** Great Debates\
**Created:** [March 5, 2014, 2:40pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928 "2014-03-05T14:40:29Z")\
**Posts on this page:** 20\
**Page:** 3

<div class="post-metadata">

**Author:** ![tim314](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/tim314/32/3468_2.png) [@tim314](https://boards.straightdope.com/u/tim314)\
**Post date:** [March 6, 2014, 5:09pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/41 "2014-03-06T17:09:36Z")

</div>

> [@Czarcasm](#):
>
> Is there any possible way for an outside agency to check to see if this is actually happening?

Well, the submitter receives the comments of the referees. If people were submitting papers to major journals and not receiving any review, word would get around. (Unless the journal is actually _writing fake comments and pretending they’re from real referees_ – but why would they? That just seems like more work for the journal.)

Perhaps with these less-legitimate journals, the submitters know they aren’t getting reviewed, and don’t care.

---

<div class="post-metadata">

**Author:** ![Melchior](https://avatars.discourse-cdn.com/v4/letter/m/e5b9ba/32.png) [@Melchior](https://boards.straightdope.com/u/Melchior)\
**Post date:** [March 6, 2014, 5:12pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/42 "2014-03-06T17:12:55Z")

</div>

> [@Budget\_Player\_Cadet](#):
>
> So this story came to my attention a while back:
> 
> [http://www.sciencemag.org/content/342/6154/60.full](http://www.sciencemag.org/content/342/6154/60.full)
> 
> One of my more “intellectually challenged” hippie friends posited that because of this, peer review was worthless and that we should ignore it. I think this is a load of crap. Peer-review is a necessary first step; all this does is show that many journals are not exactly scrupulous with it. What do you guys think?

It depends on the field and journal.

---

<div class="post-metadata">

**Author:** ![Czarcasm](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/czarcasm/32/4050_2.png) [@Czarcasm](https://boards.straightdope.com/u/Czarcasm)\
**Post date:** [March 6, 2014, 5:14pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/43 "2014-03-06T17:14:27Z")

</div>

> [@tim314](#):
>
> Well, the submitter receives the comments of the referees. If people were submitting papers to major journals and not receiving any review, word would get around. (Unless the journal is actually _writing fake comments and pretending they’re from real referees_ – but why would they? That just seems like more work for the journal.)
> 
> Perhaps with these less-legitimate journals, the submitters know they aren’t getting reviewed, and don’t care.

Not much of a hassle to hire a couple of minimum wage “reviewers” to “review” the various papers.

---

<div class="post-metadata">

**Author:** ![Amateur\_Barbarian](https://avatars.discourse-cdn.com/v4/letter/a/59ef9b/32.png) [@Amateur\_Barbarian](https://boards.straightdope.com/u/Amateur_Barbarian)\
**Post date:** [March 6, 2014, 5:19pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/44 "2014-03-06T17:19:22Z")

</div>

[QUOTE=Hector\_St\_Clare]  
I read a quote by Feyerabend once: “Prayer may not be very efficient compared to celestial mechanics, but it surely holds its own vis-à-vis some parts of economics.”   
[/QUOTE]

_Must… write that down…_ done.

---

<div class="post-metadata">

**Author:** ![Really\_Not\_All\_That\_Bright](https://avatars.discourse-cdn.com/v4/letter/r/e8c25b/32.png) [@Really\_Not\_All\_That\_Bright](https://boards.straightdope.com/u/Really_Not_All_That_Bright)\
**Post date:** [March 6, 2014, 5:27pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/45 "2014-03-06T17:27:02Z")

</div>

> [@Smeghead](#):
>
> I send a paper to a journal. An editor for that journal selects three experts in the field who would be expected to be fully conversant with the topic of my research and asks them to review my paper. They each read the paper and write up their suggestions and recommendations - they can say they think the paper should be published, or returned for minor or major corrections, or simply rejected. The editor then makes the final decision and forwards the reviewers’ comments back to me, the author. The review is anonymous. The author does not know who the reviewers are.
> 
> This is how it’s supposed to work.

Thanks! Are the reviewers’ names released if the piece is published?

---

<div class="post-metadata">

**Author:** ![Voyager](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/voyager/32/133_2.png) [@Voyager](https://boards.straightdope.com/u/Voyager)\
**Post date:** [March 6, 2014, 5:51pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/46 "2014-03-06T17:51:59Z")

</div>

> [@Hector\_St\_Clare](#):
>
> The problem with economics isn’t really that it’s non-mathematical or lacks rigor. The problem is that, unlike (say) organic chemistry or astrophysics, economics is (often) a value laden discipline, which involves normative decisions about what sort of society we want to have. Science is supposed to be value neutral, and when you start blurring the line between factual statements and normative statements, you start moving away from the realm of science.

What kind of economics have you looked at? Economic policy decisions are indeed based on political issues, but the impact of making a policy decision should not be. In fact, a major weakness of micro is the assumption of homo economicus, who makes rational decisions at all times. Behavioral economics has demonstrated that no such thing exists, and to properly predict the impact of decisions you need to properly model how people will react to them. Yes, people get involved, but that introduces more and messier inputs, and does not reduce the mathematical rigor.  
Like weather forecasting, economics forecasting is chaotic, only more so.

---

<div class="post-metadata">

**Author:** ![Voyager](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/voyager/32/133_2.png) [@Voyager](https://boards.straightdope.com/u/Voyager)\
**Post date:** [March 6, 2014, 5:58pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/47 "2014-03-06T17:58:03Z")

</div>

> [@Czarcasm](#):
>
> Not much of a hassle to hire a couple of minimum wage “reviewers” to “review” the various papers.

Only problem with that is that experts in the field will read the journal, note that many or most of the articles in it are crap, and refuse to send papers there or review for it. They’ll stop subscribing, get their library to not subscribe, and as word gets around everyone will learn that having a paper published there is approximately as useful as publishing it in stall 3 of the local restroom.

Journal quality is seen as proportional to the rejection rate. That discourages such nonsense.

---

<div class="post-metadata">

**Author:** ![Voyager](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/voyager/32/133_2.png) [@Voyager](https://boards.straightdope.com/u/Voyager)\
**Post date:** [March 6, 2014, 5:59pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/48 "2014-03-06T17:59:06Z")

</div>

> [@Really\_Not\_All\_That\_Bright](#):
>
> Thanks! Are the reviewers’ names released if the piece is published?

Not for that piece, in my experience. Reviewers for a conference are listed in one block without saying who reviewed what. I’m not sure if all journals do that.

---

<div class="post-metadata">

**Author:** ![Smeghead](https://avatars.discourse-cdn.com/v4/letter/s/f1d935/32.png) [@Smeghead](https://boards.straightdope.com/u/Smeghead)\
**Post date:** [March 6, 2014, 6:03pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/49 "2014-03-06T18:03:01Z")

</div>

> [@Czarcasm](#):
>
> Is there any possible way for an outside agency to check to see if this is actually happening?

It’s sort of self-correcting. If it’s not happening properly, the quality of the papers will drop, and people will notice. If something fishy gets published in a supposedly high-quality journal, word gets around, and it can be an embarrassment to the journal. The problem is not with the actual good journals - they’re all doing it right. It’s the proliferation of these “journals” that wrap themselves up in the veneer of respectability to hide the fact that they’re garbage. People who work in the field generally know the difference, but the general public and especially the media can be fooled.

Consider the morons who recently claimed to have “published” their Bigfoot genome in a peer-reviewed journal. Which, it turned out, they had founded. And which had published exactly one paper.

> [@Really\_Not\_All\_That\_Bright](#):
>
> Thanks! Are the reviewers’ names released if the piece is published?

Nope. It’s kept anonymous to encourage reviewers to comment honestly.

---

<div class="post-metadata">

**Author:** ![yellowjacketcoder](https://avatars.discourse-cdn.com/v4/letter/y/4bbf92/32.png) [@yellowjacketcoder](https://boards.straightdope.com/u/yellowjacketcoder)\
**Post date:** [March 6, 2014, 6:11pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/50 "2014-03-06T18:11:54Z")

</div>

> [@Really\_Not\_All\_That\_Bright](#):
>
> Thanks! Are the reviewers’ names released if the piece is published?

When I was in grad school, no they were not (it may be different for other fields).

Although, the idea that the reviewers are anonymous is really an ideal and not reality. In many fields, the number of qualified reviewers is so small, that you get to learn their writing style and the kind of suggestions they make. When I got reviews back for my first paper, the postdoc helping me out was able to identify all three reviewers in under a minute, between writing style, suggestions of works to cite, etc.

---

<div class="post-metadata">

**Author:** ![Really\_Not\_All\_That\_Bright](https://avatars.discourse-cdn.com/v4/letter/r/e8c25b/32.png) [@Really\_Not\_All\_That\_Bright](https://boards.straightdope.com/u/Really_Not_All_That_Bright)\
**Post date:** [March 6, 2014, 6:21pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/51 "2014-03-06T18:21:06Z")

</div>

> [@Smeghead](#):
>
> Nope. It’s kept anonymous to encourage reviewers to comment honestly.

Seems like it might need to be made transparent to encourage reviewers to actually read things.

---

<div class="post-metadata">

**Author:** ![Amateur\_Barbarian](https://avatars.discourse-cdn.com/v4/letter/a/59ef9b/32.png) [@Amateur\_Barbarian](https://boards.straightdope.com/u/Amateur_Barbarian)\
**Post date:** [March 6, 2014, 6:34pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/52 "2014-03-06T18:34:37Z")

</div>

> [@Voyager](#):
>
> Yes, people get involved, but that introduces more and messier inputs, and does not reduce the mathematical rigor.

Without pushing this hijack too much further, that’s precisely the problem: you can have mathematical rigor like a skyscraper’s frame and still have zero objective value. There is, IMHO/IME, in too many fields, a sense that elaborate mathematical exercises transform subjectivity and vagueness into rigorous results. It’s GIGO, or more accurately FIFO (fuzziness in…) and no page of equations can change that.

“Jane is a woman. Women are brainless prats. Therefore Jane is a brainless prat.” - absolutely rigorous logical sequence, unchallengeable in terms of logic. But complete nonsense in every other way (starting with the fact that no one names their kids Jane any more). Economics sets its own precepts and rules, orders and analyzes selective data according to those rules, and - guess what! - has an internally rigorous framework! Yay!

---

<div class="post-metadata">

**Author:** ![Voyager](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/voyager/32/133_2.png) [@Voyager](https://boards.straightdope.com/u/Voyager)\
**Post date:** [March 6, 2014, 7:03pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/53 "2014-03-06T19:03:53Z")

</div>

> [@yellowjacketcoder](#):
>
> When I was in grad school, no they were not (it may be different for other fields).
> 
> Although, the idea that the reviewers are anonymous is really an ideal and not reality. In many fields, the number of qualified reviewers is so small, that you get to learn their writing style and the kind of suggestions they make. When I got reviews back for my first paper, the postdoc helping me out was able to identify all three reviewers in under a minute, between writing style, suggestions of works to cite, etc.

I understand that in some fields the author is asked to recommend reviewers. Other reviewers are added by the editor. Then you’d have a clue. Such a system is not used in the journals and conferences I’m involved with.

---

<div class="post-metadata">

**Author:** ![Voyager](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/voyager/32/133_2.png) [@Voyager](https://boards.straightdope.com/u/Voyager)\
**Post date:** [March 6, 2014, 7:09pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/54 "2014-03-06T19:09:46Z")

</div>

> [@Amateur\_Barbarian](#):
>
> Without pushing this hijack too much further, that’s precisely the problem: you can have mathematical rigor like a skyscraper’s frame and still have zero objective value. There is, IMHO/IME, in too many fields, a sense that elaborate mathematical exercises transform subjectivity and vagueness into rigorous results. It’s GIGO, or more accurately FIFO (fuzziness in…) and no page of equations can change that.

Let’s forget economics and look at this. This is exactly a flaw in lots of work - I’ve seen it in engineering also. In fact I devoted some slides in my Engineering economics tutorial to this fallacy. I’ve seen people graph two very noisy curves, plot the intersection, and say that is exactly where you should decide the method is worthwhile or not worthwhile. Drives me nuts.

---

<div class="post-metadata">

**Author:** ![Amateur\_Barbarian](https://avatars.discourse-cdn.com/v4/letter/a/59ef9b/32.png) [@Amateur\_Barbarian](https://boards.straightdope.com/u/Amateur_Barbarian)\
**Post date:** [March 6, 2014, 7:21pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/55 "2014-03-06T19:21:00Z")

</div>

> [@Voyager](#):
>
> Let’s forget economics and look at this. This is exactly a flaw in lots of work - I’ve seen it in engineering also. In fact I devoted some slides in my Engineering economics tutorial to this fallacy. I’ve seen people graph two very noisy curves, plot the intersection, and say that is exactly where you should decide the method is worthwhile or not worthwhile. Drives me nuts.

Years ago I saw an argument that successive generations of engineers were relying more and more on computer-based calculation and modeling, and thus losing touch with the “meat” of their specialty. That is, an old-school civil engineer could look at a blueprint and know in his gut that something was wrong, whereas a newer graduate would trust the automation and have no sense of the problem.

You can push that argument around a lot of ways, but I have enough old-school experience in a few things, and have worked with enough thick-knuckled old engineers and builders, AND with enough geek engineers, and have seen both miraculous cases of “Wait, something’s wrong” and smug “It’s all been analyzed” to believe the gist of it.

I can see that such thinking - and training that emphasizes only advanced tools - could lead to a misunderstanding of how precise calculations map to a real world solution.

---

<div class="post-metadata">

**Author:** ![Smeghead](https://avatars.discourse-cdn.com/v4/letter/s/f1d935/32.png) [@Smeghead](https://boards.straightdope.com/u/Smeghead)\
**Post date:** [March 6, 2014, 7:36pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/56 "2014-03-06T19:36:09Z")

</div>

> [@Really\_Not\_All\_That\_Bright](#):
>
> Seems like it might need to be made transparent to encourage reviewers to actually read things.

That’s supposed to be part of the job of the editorial board - keep track of “good” reviewers and “bad”. But really, this whole thing is a massive volunteer effort. People chip in because it’s expected as part of being a useful member of the scientific community. It’s part of the culture. I don’t think that trying to impose massive oversight, sets of rules, rankings, and box-checking is ever going to work. It’s not a perfect system by any means, but despite stories like the ones that started this thread, it’s pretty darned good.

---

<div class="post-metadata">

**Author:** ![Really\_Not\_All\_That\_Bright](https://avatars.discourse-cdn.com/v4/letter/r/e8c25b/32.png) [@Really\_Not\_All\_That\_Bright](https://boards.straightdope.com/u/Really_Not_All_That_Bright)\
**Post date:** [March 6, 2014, 7:57pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/57 "2014-03-06T19:57:31Z")

</div>

You don’t get paid for peer review?

---

<div class="post-metadata">

**Author:** ![Voyager](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/voyager/32/133_2.png) [@Voyager](https://boards.straightdope.com/u/Voyager)\
**Post date:** [March 6, 2014, 8:02pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/58 "2014-03-06T20:02:37Z")

</div>

> [@Really\_Not\_All\_That\_Bright](#):
>
> You don’t get paid for peer review?

We wish!

---

<div class="post-metadata">

**Author:** ![mozchron](https://avatars.discourse-cdn.com/v4/letter/m/c2a13f/32.png) [@mozchron](https://boards.straightdope.com/u/mozchron)\
**Post date:** [March 6, 2014, 8:03pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/59 "2014-03-06T20:03:55Z")

</div>

> [@Really\_Not\_All\_That\_Bright](#):
>
> You don’t get paid for peer review?

Heh. Good one.

---

<div class="post-metadata">

**Author:** ![Algher](https://avatars.discourse-cdn.com/v4/letter/a/3d9bf3/32.png) [@Algher](https://boards.straightdope.com/u/Algher)\
**Post date:** [March 6, 2014, 10:45pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/60 "2014-03-06T22:45:32Z")

</div>

> [@Voyager](#):
>
> Not for that piece, in my experience. Reviewers for a conference are listed in one block without saying who reviewed what. I’m not sure if all journals do that.

Reviews in my wife’s field are anonymous. However, after awhile you can tell who is reviewing you by the comments.

[Previous page](https://boards.straightdope.com/t/is-peer-review-valid/682928.md?page=2)

[Next page](https://boards.straightdope.com/t/is-peer-review-valid/682928.md?page=4)
