# Is Peer-Review Valid?

**URL:** <https://boards.straightdope.com/t/is-peer-review-valid/682928>\
**Category:** Great Debates\
**Created:** [March 5, 2014, 2:40pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928 "2014-03-05T14:40:29Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![outlierrn](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/outlierrn/32/3005_2.png) [@outlierrn](https://boards.straightdope.com/u/outlierrn)\
**Post date:** [March 6, 2014, 12:40am UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/21 "2014-03-06T00:40:33Z")

</div>

It’s a tool; a useful, imperfect tool. Of course, we should keep our ignorance fighting glasses near to hand, but did your friend have a better tool to replace it with?

---

<div class="post-metadata">

**Author:** ![Amateur\_Barbarian](https://avatars.discourse-cdn.com/v4/letter/a/59ef9b/32.png) [@Amateur\_Barbarian](https://boards.straightdope.com/u/Amateur_Barbarian)\
**Post date:** [March 6, 2014, 12:42am UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/22 "2014-03-06T00:42:39Z")

</div>

> [@Voyager](#):
>
> Depends on the kind of psychology you do. The kind my daughter does is mathematically driven. Statistical validity of experimental results is extremely important. Not all psychology is like that.

I have to tread carefully here, as the Mrs. is a behaviorist, but the presence of math and stat is not scientific validity _per se_.

After all, what impresses many people about astrology is all the precise numbers and formulae. I think it was Sylvester Stallone who was gushing about his mother’s fascination with astrology and said, “It’'s all mathematical and stuff.”

Then there’s economics, which is a closed universe made up of 50% numbers and 50% self-referential interpretations of them…

Just sayin’.

---

<div class="post-metadata">

**Author:** ![Smeghead](https://avatars.discourse-cdn.com/v4/letter/s/f1d935/32.png) [@Smeghead](https://boards.straightdope.com/u/Smeghead)\
**Post date:** [March 6, 2014, 12:54am UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/23 "2014-03-06T00:54:20Z")

</div>

Having just finished contributing to a peer review, I feel I can honestly say that we combed that paper with a fine-toothed comb, and suspect that the authors are going to feel like they got hit with a sledgehammer.

And it was a pretty good paper, to be honest.

---

<div class="post-metadata">

**Author:** ![CalMeacham](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/calmeacham/32/35_2.png) [@CalMeacham](https://boards.straightdope.com/u/CalMeacham)\
**Post date:** [March 6, 2014, 12:57am UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/24 "2014-03-06T00:57:08Z")

</div>

Peer Review is a good thing, and in my experience has been pretty thorough (I’ve had papers come back with suggestions, and even rejected).

This is not to say it is invariably the case. There have been other cases of outrageous papers making it through the gauntlet and emerged without any problems, most notably the Sokal case. There was another one in Eastern Europe this past year. ( [A Serbian Sokal? Authors spoof pub with Ron Jeremy and Michael Jackson references – Retraction Watch](http://retractionwatch.wordpress.com/2013/09/23/a-serbian-sokal-authors-spoof-pub-with-ron-jeremy-and-michael-jackson-references/#more-15762) )

Even worse, some papers that were not deliberate outrageous fakes have passed through with ludicrous assertions that were later caught by other scientists. I was on a panel discussing these two months ago.  
For instance, the photographs in one paper appear to have been ludricrously badly photoshopped:

: [http://www.chemistry-blog.com/2013/08/13/alleged-data-manipulation-in-nano-letters-and-acs-nano-from-the-pease-group/](http://www.chemistry-blog.com/2013/08/13/alleged-data-manipulation-in-nano-letters-and-acs-nano-from-the-pease-group/)

: [http://retractionwatch.wordpress.com/2013/08/16/nano-letters-retracts-chopstick-nanorod-paper-questioned-this-week-on-chemistry-blogs/](http://retractionwatch.wordpress.com/2013/08/16/nano-letters-retracts-chopstick-nanorod-paper-questioned-this-week-on-chemistry-blogs/)

: [Chemjobber: Does this picture look weird?](http://chemjobber.blogspot.com/2013/08/does-this-picture-look-weird.html)

Or consider this paper, with its weird statement left in the appendix:

[http://www.chemistry-blog.com/2013/08/07/when-authors-forget-to-fake-an-elemental-analysis/](http://www.chemistry-blog.com/2013/08/07/when-authors-forget-to-fake-an-elemental-analysis/)

---

<div class="post-metadata">

**Author:** ![Agnostic\_Pagan](https://avatars.discourse-cdn.com/v4/letter/a/c4cdca/32.png) [@Agnostic\_Pagan](https://boards.straightdope.com/u/Agnostic_Pagan)\
**Post date:** [March 6, 2014, 2:07am UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/25 "2014-03-06T02:07:51Z")

</div>

The only flaw in the process I see is a lack of certification. Peer-reviewers should be required to meet some standard in order to review papers. The fact that a reviewer may have a PhD or similar degree does not seem to be sufficient. And journals should be audited that their review process also meets some standards, and let them put a ‘seal of approval’ on their cover. It is a bit surprising that this has not occurred already, or if it has, no one seems to be aware of such.

Each discipline should be able to set its own standards, but let the overall process be managed by the ISO or similar organization.

The publish or perish model also needs to be changed. I would rather have two or three books published by a researcher that shows the culmination of several years work, than twenty papers published annually that might tie together. That latter process might be valid for some lines of research, but not sure it is true for most. I know that some scholars do that - publish books summing up their previous research - but it appears to be the exception than the rule.

But that would probably require a serious change in how to fund science away from the grant model. I do not know what would be a better model, but the current system seems to be breaking down. Perhaps a certification process could reinvigorate it and increase its utility. If I was on a grant-making board, I would certainly give ‘certified’ researchers greater consideration, just as when hiring any other professional.

---

<div class="post-metadata">

**Author:** ![mozchron](https://avatars.discourse-cdn.com/v4/letter/m/c2a13f/32.png) [@mozchron](https://boards.straightdope.com/u/mozchron)\
**Post date:** [March 6, 2014, 3:30am UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/26 "2014-03-06T03:30:09Z")

</div>

> [@Hentor\_the\_Barbarian](#):
>
> Seconded. This is about open access journals, which are shit. It isn’t an indictment of the peer review process.

No, _SOME_ Open Access Journals are shit. This one is shit, and published by a shit publisher.

The PLoS journals, BMC journals, Scientific Reports (Published by Nature publishing), eLife, PeerJ, and others have high standards and in some cases very high Impact Factors.

My lab is moving toward the goal of 100% Open Access publishing. But there is Open Access and Open Access. You need to be careful and only publish with established and reputable organizations.

---

<div class="post-metadata">

**Author:** ![mozchron](https://avatars.discourse-cdn.com/v4/letter/m/c2a13f/32.png) [@mozchron](https://boards.straightdope.com/u/mozchron)\
**Post date:** [March 6, 2014, 3:32am UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/27 "2014-03-06T03:32:24Z")

</div>

> [@Agnostic\_Pagan](#):
>
> The publish or perish model also needs to be changed. I would rather have two or three books published by a researcher that shows the culmination of several years work, than twenty papers published annually that might tie together. That latter process might be valid for some lines of research, but not sure it is true for most.

Off topic, but as a working scientist this is an incredibly stupid statement, showing little to no understanding of how scientific research is typically performed.

---

<div class="post-metadata">

**Author:** ![Voyager](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/voyager/32/133_2.png) [@Voyager](https://boards.straightdope.com/u/Voyager)\
**Post date:** [March 6, 2014, 5:55am UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/28 "2014-03-06T05:55:00Z")

</div>

> [@Amateur\_Barbarian](#):
>
> I have to tread carefully here, as the Mrs. is a behaviorist, but the presence of math and stat is not scientific validity _per se_.
> 
> After all, what impresses many people about astrology is all the precise numbers and formulae. I think it was Sylvester Stallone who was gushing about his mother’s fascination with astrology and said, “It’'s all mathematical and stuff.”
> 
> Then there’s economics, which is a closed universe made up of 50% numbers and 50% self-referential interpretations of them…
> 
> Just sayin’.

What is the significance of the result of an astrological prediction? They have been tested - and failed. Let’s just say her favorite Christmas present one year was a Matlab manual.

As for economics, have you ever taken econometrics? It is nastier than any math I took as an engineer at MIT. The problem is not the models or the math, it is the inability to conduct a controlled experiment.

---

<div class="post-metadata">

**Author:** ![Voyager](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/voyager/32/133_2.png) [@Voyager](https://boards.straightdope.com/u/Voyager)\
**Post date:** [March 6, 2014, 6:10am UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/29 "2014-03-06T06:10:52Z")

</div>

> [@Agnostic\_Pagan](#):
>
> The only flaw in the process I see is a lack of certification. Peer-reviewers should be required to meet some standard in order to review papers. The fact that a reviewer may have a PhD or similar degree does not seem to be sufficient. And journals should be audited that their review process also meets some standards, and let them put a ‘seal of approval’ on their cover. It is a bit surprising that this has not occurred already, or if it has, no one seems to be aware of such.

How do you propose to audit reviewers? People don’t pop out of the blue to do reviews, they get selected by the editor. In my experience (and I’ve read hundreds of reviews as an editor and program chair) the worst reviews are by the most senior people, who are busy and say things like “good job.” Grad students really care.  
In our conference we track reviewers average scores in previous years, so that reviews of a 4 from a reviewer who averages 3 doesn’t kill a paper, and a review of 8 from a reviewer who averages 9 doesn’t help it.

> [@](#):
>
> The publish or perish model also needs to be changed. I would rather have two or three books published by a researcher that shows the culmination of several years work, than twenty papers published annually that might tie together. That latter process might be valid for some lines of research, but not sure it is true for most. I know that some scholars do that - publish books summing up their previous research - but it appears to be the exception than the rule.

Are you aware that books are not peer reviewed? Many publishers have an expert review the book proposal and chapters, but only one person, and you clearly can’t do the kind of job you do with a paper. (I’ve done a few - you even get paid. 🙂 ) Plus the publisher is on deadline, hopes to make money, and there is no competition.  
BTW after my adviser died the one I got to replace him was writing a book instead of writing papers, where another professor in the same field was writing papers. One reason I switched universities was I knew my professor was not going to get tenure - I was right. So I don’t think you understand the process very well.

> [@](#):
>
> But that would probably require a serious change in how to fund science away from the grant model. I do not know what would be a better model, but the current system seems to be breaking down. Perhaps a certification process could reinvigorate it and increase its utility. If I was on a grant-making board, I would certainly give ‘certified’ researchers greater consideration, just as when hiring any other professional.

Have you ever done grant review? Didn’t think so. A big part of the proposal is qualifications. Not to mention that almost all the grants in the real NSF reviewing I did (before I got stuck with small business crap) were from people I and the rest of the panel knew personally and by reputation. If there was a certification process it would probably be done by some of the same people reviewing the grant.

Most areas are small when you get into the specialties. In fact, there is a move in some areas to remove all names from papers so that there is no halo effect.

---

<div class="post-metadata">

**Author:** ![Voyager](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/voyager/32/133_2.png) [@Voyager](https://boards.straightdope.com/u/Voyager)\
**Post date:** [March 6, 2014, 6:12am UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/30 "2014-03-06T06:12:48Z")

</div>

> [@mozchron](#):
>
> Off topic, but as a working scientist this is an incredibly stupid statement, showing little to no understanding of how scientific research is typically performed.

Nor the target audience for a book, nor the circulation figures of the best specialized journals.

---

<div class="post-metadata">

**Author:** ![DataX](https://avatars.discourse-cdn.com/v4/letter/d/e99b99/32.png) [@DataX](https://boards.straightdope.com/u/DataX)\
**Post date:** [March 6, 2014, 7:38am UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/31 "2014-03-06T07:38:47Z")

</div>

> [@CalMeacham](#):
>
> Peer Review is a good thing, and in my experience has been pretty thorough (I’ve had papers come back with suggestions, and even rejected).
> 
> This is not to say it is invariably the case. There have been other cases of outrageous papers making it through the gauntlet and emerged without any problems, most notably the Sokal case. There was another one in Eastern Europe this past year. ( [A Serbian Sokal? Authors spoof pub with Ron Jeremy and Michael Jackson references – Retraction Watch](http://retractionwatch.wordpress.com/2013/09/23/a-serbian-sokal-authors-spoof-pub-with-ron-jeremy-and-michael-jackson-references/#more-15762) )
> 
> Even worse, some papers that were not deliberate outrageous fakes have passed through with ludicrous assertions that were later caught by other scientists. I was on a panel discussing these two months ago.  
> For instance, the photographs in one paper appear to have been ludricrously badly photoshopped:
> 
> : [http://www.chemistry-blog.com/2013/08/13/alleged-data-manipulation-in-nano-letters-and-acs-nano-from-the-pease-group/](http://www.chemistry-blog.com/2013/08/13/alleged-data-manipulation-in-nano-letters-and-acs-nano-from-the-pease-group/)
> 
> : [Nano Letters retracts chopstick nanorod paper questioned this week on chemistry blogs – Retraction Watch](http://retractionwatch.wordpress.com/2013/08/16/nano-letters-retracts-chopstick-nanorod-paper-questioned-this-week-on-chemistry-blogs/)
> 
> : [Chemjobber: Does this picture look weird?](http://chemjobber.blogspot.com/2013/08/does-this-picture-look-weird.html)
> 
> Or consider this paper, with its weird statement left in the appendix:
> 
> [http://www.chemistry-blog.com/2013/08/07/when-authors-forget-to-fake-an-elemental-analysis/](http://www.chemistry-blog.com/2013/08/07/when-authors-forget-to-fake-an-elemental-analysis/)

Wow - thanks for that - really made me smile. The person who photoshopped that has gigantic testicles (or ovaries). I just can’t imagine the confidence you must feel to just edit something like that - blithely unaware how obvious it appears to anyone who has every tried to - well Photoshop - anything.

Imagine what their 1040 (or country equivalent) must look like.

---

<div class="post-metadata">

**Author:** ![Hentor\_the\_Barbarian](https://avatars.discourse-cdn.com/v4/letter/h/8edcca/32.png) [@Hentor\_the\_Barbarian](https://boards.straightdope.com/u/Hentor_the_Barbarian)\
**Post date:** [March 6, 2014, 12:39pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/32 "2014-03-06T12:39:00Z")

</div>

> [@mozchron](#):
>
> No, _SOME_ Open Access Journals are shit. This one is shit, and published by a shit publisher.
> 
> The PLoS journals, BMC journals, Scientific Reports (Published by Nature publishing), eLife, PeerJ, and others have high standards and in some cases very high Impact Factors.
> 
> My lab is moving toward the goal of 100% Open Access publishing. But there is Open Access and Open Access. You need to be careful and only publish with established and reputable organizations.

Fair enough, I haven’t evaluated all Open Access journals.

My comment was based on my experience of having received a dozen or so requests to serve on the editorial boards of open access journals that were in topics I have no knowledge of, and the fact that I receive on a seemingly weekly basis an invitation to submit a manuscript to a journal that, again, is on a topic that is way outside of anything in which I would be seen as an expert.

I did agree to peer review for an open access journal once, and the manuscript was below the level I would expect from a student’s work on a course paper.

---

<div class="post-metadata">

**Author:** ![Hector\_St\_Clare](https://avatars.discourse-cdn.com/v4/letter/h/b5ac83/32.png) [@Hector\_St\_Clare](https://boards.straightdope.com/u/Hector_St_Clare)\
**Post date:** [March 6, 2014, 3:10pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/33 "2014-03-06T15:10:48Z")

</div>

> [@Voyager](#):
>
> What is the significance of the result of an astrological prediction? They have been tested - and failed. Let’s just say her favorite Christmas present one year was a Matlab manual.
> 
> As for economics, have you ever taken econometrics? It is nastier than any math I took as an engineer at MIT. The problem is not the models or the math, it is the inability to conduct a controlled experiment.

The problem with economics isn’t really that it’s non-mathematical or lacks rigor. The problem is that, unlike (say) organic chemistry or astrophysics, economics is (often) a value laden discipline, which involves normative decisions about what sort of society we want to have. Science is supposed to be value neutral, and when you start blurring the line between factual statements and normative statements, you start moving away from the realm of science.

And yes, it’s of course true that economics (as well as history, anthropology, etc.) lack the ability to do controlled experiments. Societies aren’t independent replicates, they all influence each other, and it’s very difficult to control all the differences between them.

---

<div class="post-metadata">

**Author:** ![Really\_Not\_All\_That\_Bright](https://avatars.discourse-cdn.com/v4/letter/r/e8c25b/32.png) [@Really\_Not\_All\_That\_Bright](https://boards.straightdope.com/u/Really_Not_All_That_Bright)\
**Post date:** [March 6, 2014, 3:14pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/34 "2014-03-06T15:14:12Z")

</div>

Can somebody give me the whistle-stop version of how the peer review process works for publication?

---

<div class="post-metadata">

**Author:** ![Amateur\_Barbarian](https://avatars.discourse-cdn.com/v4/letter/a/59ef9b/32.png) [@Amateur\_Barbarian](https://boards.straightdope.com/u/Amateur_Barbarian)\
**Post date:** [March 6, 2014, 3:34pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/35 "2014-03-06T15:34:32Z")

</div>

> [@Voyager](#):
>
> What is the significance of the result of an astrological prediction?  
> […]  
> As for economics, have you ever taken econometrics? It is nastier than any math I took as an engineer at MIT.

So… one’s all mathy but completely invalid, and the other is all mathy but… completely subjective.

That mathy stuff. You can do anything with it. 😃

---

<div class="post-metadata">

**Author:** ![Amateur\_Barbarian](https://avatars.discourse-cdn.com/v4/letter/a/59ef9b/32.png) [@Amateur\_Barbarian](https://boards.straightdope.com/u/Amateur_Barbarian)\
**Post date:** [March 6, 2014, 3:37pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/36 "2014-03-06T15:37:28Z")

</div>

> [@Hector\_St\_Clare](#):
>
> The problem with economics isn’t really that it’s non-mathematical or lacks rigor.

Well, no, the problem is that it’s reliant on intensive mathematical modeling of entirely selective and subjective concepts. Deeply analyzed circular reasoning with less foundation than some forms of astrology.

---

<div class="post-metadata">

**Author:** ![Smeghead](https://avatars.discourse-cdn.com/v4/letter/s/f1d935/32.png) [@Smeghead](https://boards.straightdope.com/u/Smeghead)\
**Post date:** [March 6, 2014, 3:41pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/37 "2014-03-06T15:41:51Z")

</div>

> [@Really\_Not\_All\_That\_Bright](#):
>
> Can somebody give me the whistle-stop version of how the peer review process works for publication?

I send a paper to a journal. An editor for that journal selects three experts in the field who would be expected to be fully conversant with the topic of my research and asks them to review my paper. They each read the paper and write up their suggestions and recommendations - they can say they think the paper should be published, or returned for minor or major corrections, or simply rejected. The editor then makes the final decision and forwards the reviewers’ comments back to me, the author. The review is anonymous. The author does not know who the reviewers are.

This is how it’s supposed to work.

---

<div class="post-metadata">

**Author:** ![Czarcasm](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/czarcasm/32/4050_2.png) [@Czarcasm](https://boards.straightdope.com/u/Czarcasm)\
**Post date:** [March 6, 2014, 3:46pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/38 "2014-03-06T15:46:51Z")

</div>

> [@Smeghead](#):
>
> I send a paper to a journal. An editor for that journal selects three experts in the field who would be expected to be fully conversant with the topic of my research and asks them to review my paper. They each read the paper and write up their suggestions and recommendations - they can say they think the paper should be published, or returned for minor or major corrections, or simply rejected. The editor then makes the final decision and forwards the reviewers’ comments back to me, the author. The review is anonymous. The author does not know who the reviewers are.
> 
> This is how it’s supposed to work.

Is there any possible way for an outside agency to check to see if this is actually happening?

---

<div class="post-metadata">

**Author:** ![Hector\_St\_Clare](https://avatars.discourse-cdn.com/v4/letter/h/b5ac83/32.png) [@Hector\_St\_Clare](https://boards.straightdope.com/u/Hector_St_Clare)\
**Post date:** [March 6, 2014, 3:47pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/39 "2014-03-06T15:47:14Z")

</div>

> [@Amateur\_Barbarian](#):
>
> Well, no, the problem is that it’s reliant on intensive mathematical modeling of entirely selective and subjective concepts. Deeply analyzed circular reasoning with less foundation than some forms of astrology.

I read a quote by Feyerabend once: “Prayer may not be very efficient compared to celestial mechanics, but it surely holds its own vis-à-vis some parts of economics.”

---

<div class="post-metadata">

**Author:** ![mozchron](https://avatars.discourse-cdn.com/v4/letter/m/c2a13f/32.png) [@mozchron](https://boards.straightdope.com/u/mozchron)\
**Post date:** [March 6, 2014, 4:40pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/40 "2014-03-06T16:40:24Z")

</div>

> [@Hentor\_the\_Barbarian](#):
>
> Fair enough, I haven’t evaluated all Open Access journals.
> 
> My comment was based on my experience of having received a dozen or so requests to serve on the editorial boards of open access journals that were in topics I have no knowledge of, and the fact that I receive on a seemingly weekly basis an invitation to submit a manuscript to a journal that, again, is on a topic that is way outside of anything in which I would be seen as an expert.
> 
> I did agree to peer review for an open access journal once, and the manuscript was below the level I would expect from a student’s work on a course paper.

You’ve probably been spammed by Hindawi, Bentham, OMICS or one of the other numerous scam outfits operating (mostly) out of India. These aren’t Open Access Journals. They are scam vanity presses masquerading as journals.

A good list can be found here: [http://scholarlyoa.com/2012/12/06/bealls-list-of-predatory-publishers-2013/](http://scholarlyoa.com/2012/12/06/bealls-list-of-predatory-publishers-2013/)

Open Access has NOTHING to do with Peer Review. All it means is that readers have access to the articles without having to pay a fee. No more, no less.

[Previous page](https://boards.straightdope.com/t/is-peer-review-valid/682928.md?page=1)

[Next page](https://boards.straightdope.com/t/is-peer-review-valid/682928.md?page=3)
