# Is Peer-Review Valid? **URL:** \ **Category:** Great Debates\ **Created:** [March 5, 2014, 2:40pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928 "2014-03-05T14:40:29Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Budget\_Player\_Cadet](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/budget_player_cadet/32/205_2.png) [@Budget\_Player\_Cadet](https://boards.straightdope.com/u/Budget_Player_Cadet)\ **Post date:** [March 5, 2014, 2:40pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/1 "2014-03-05T14:40:29Z")
So this story came to my attention a while back: [http://www.sciencemag.org/content/342/6154/60.full](http://www.sciencemag.org/content/342/6154/60.full) > [@](#): > > On 4 July, good news arrived in the inbox of Ocorrafoo Cobange, a biologist at the Wassee Institute of Medicine in Asmara. It was the official letter of acceptance for a paper he had submitted 2 months earlier to the Journal of Natural Pharmaceuticals, describing the anticancer properties of a chemical that Cobange had extracted from a lichen. > > In fact, it should have been promptly rejected. Any reviewer with more than a high-school knowledge of chemistry and the ability to understand a basic data plot should have spotted the paper’s short-comings immediately. Its experiments are so hopelessly flawed that the results are meaningless. > > I know because I wrote the paper. Ocorrafoo Cobange does not exist, nor does the Wassee Institute of Medicine. Over the past 10 months, I have submitted 304 versions of the wonder drug paper to open-access journals. More than half of the journals accepted the paper, failing to notice its fatal flaws. Beyond that headline result, the data from this sting operation reveal the contours of an emerging Wild West in academic publishing. One of my more “intellectually challenged” hippie friends posited that because of this, peer review was worthless and that we should ignore it. I think this is a load of crap. Peer-review is a necessary first step; all this does is show that many journals are not exactly scrupulous with it. What do you guys think? ---
**Author:** ![Amateur\_Barbarian](https://avatars.discourse-cdn.com/v4/letter/a/59ef9b/32.png) [@Amateur\_Barbarian](https://boards.straightdope.com/u/Amateur_Barbarian)\ **Post date:** [March 5, 2014, 2:44pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/2 "2014-03-05T14:44:45Z")
The explosion of journals (from around one per major field in the 1950s-60s) and meteoric rise of publish-at-any-cost (used to be publish or perish; now it’s publish or never get another job in this town, kid) have diluted the nature of peer-reviewed publication, yes. It’s particularly bad in the money-driven fields like pharma. All fields would be interested to hear of an alternative system. Got one? (Me, I’d like to see a lower bar established, to eliminate the practice of ten minor and nearly meaningless publications in place of one or two that actually advance knowledge and understanding in the field. But nobody listen to Zathras.) ---
**Author:** ![Czarcasm](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/czarcasm/32/4050_2.png) [@Czarcasm](https://boards.straightdope.com/u/Czarcasm)\ **Post date:** [March 5, 2014, 3:12pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/3 "2014-03-05T15:12:51Z")
How do the older established peer-reviewed journals stack up against the new ones when it comes to reliability? I noticed that the Journal of Natural Pharmaceuticals is less than 4 years old. ---
**Author:** ![DrCube](https://avatars.discourse-cdn.com/v4/letter/d/a3d4f5/32.png) [@DrCube](https://boards.straightdope.com/u/DrCube)\ **Post date:** [March 5, 2014, 3:18pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/4 "2014-03-05T15:18:47Z")
There’s a difference between “peer review” as it is practiced by publishers, and the fact that scientists the world over spend a lot of time replicating important experiments and verifying contending theories. Any particular paper has a good chance of being bullshit. But in ten or twenty years, when that paper’s main idea has been fleshed out, tested repeatedly, and come to be the consensus view, you can bet that it’s very likely correct. Or as close as you get in science. But to a certain extent, whenever you see a news story that begins with “Studies show…”, there’s a good chance it’s crap. ---
**Author:** ![Really\_Not\_All\_That\_Bright](https://avatars.discourse-cdn.com/v4/letter/r/e8c25b/32.png) [@Really\_Not\_All\_That\_Bright](https://boards.straightdope.com/u/Really_Not_All_That_Bright)\ **Post date:** [March 5, 2014, 3:21pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/5 "2014-03-05T15:21:36Z")
> [@Czarcasm](#): > > How do the older established peer-reviewed journals stack up against the new ones when it comes to reliability? I noticed that the Journal of Natural Pharmaceuticals is less than 4 years old. Its [website](http://www.jnatpharm.org/) certainly hasn’t been reviewed (by peers or otherwise): > [@](#): > > Journal of Natural Pharmaceuticals (JNP) (Print ISSN: 2229-5119, E-ISSN: 2229-7715) is peer-reviewed journal published on behalf of [Phcog.Net](http://Phcog.Net). The journal publishes information relating to Pharmacognosy, Natural Products, Phytochemistry, Phytopharmacology. The Journal is published Semiannual in June and December. ---
**Author:** ![tim314](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/tim314/32/3468_2.png) [@tim314](https://boards.straightdope.com/u/tim314)\ **Post date:** [March 5, 2014, 6:47pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/6 "2014-03-05T18:47:14Z")
> [@DrCube](#): > > There’s a difference between “peer review” as it is practiced by publishers, and the fact that scientists the world over spend a lot of time replicating important experiments and verifying contending theories. > > Any particular paper has a good chance of being bullshit. But in ten or twenty years, when that paper’s main idea has been fleshed out, tested repeatedly, and come to be the consensus view, you can bet that it’s very likely correct. Or as close as you get in science. > > But to a certain extent, whenever you see a news story that begins with “Studies show…”, there’s a good chance it’s crap. This is all true, and is important to keep in mind when reading news of scientific research. The fact that something appeared in a peer-reviewed journal doesn’t in any way make it correct. But, I think this is also a little beside the point. The point of peer review isn’t to validate the research is _correct_. To do that, the reviewer would have to actually try to reproduce the results of the experiment. The point of peer review is, roughly speaking, to validate the quality of the paper being submitted. Are the statements contained within it coherent and expressed in a comprehensible way? Is it free of obvious flaws and errors? Is the research significantly noteworthy to be worthy of inclusion in this journal? That sort of thing. If a paper is passes peer-review, and its conclusions are later proven wrong by subsequent research, that’s no failing of the review system. But if a paper which is nonsense, or obviously fraudulent, somehow makes it into a peer-reviewed journal, then that absolutely is a failing of the peer-review system, and suggests that the reviewers were rubber stamping things without bothering to read or understand them. (Edited to add: Or it suggests that the publisher is lying when claiming they had the paper reviewed by experts in the field.) ---
**Author:** ![tim314](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/tim314/32/3468_2.png) [@tim314](https://boards.straightdope.com/u/tim314)\ **Post date:** [March 5, 2014, 6:51pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/7 "2014-03-05T18:51:43Z")
Regarding the question in the subject, some form of quality control by academic journals seems like a must. I think probably peer-review is still the best method of quality control, but no doubt it isn’t perfect and there is room for improvement. Doing away with peer review without a suitable replacement doesn’t make much sense… ---
**Author:** ![Deeg](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/deeg/32/2955_2.png) [@Deeg](https://boards.straightdope.com/u/Deeg)\ **Post date:** [March 5, 2014, 6:53pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/8 "2014-03-05T18:53:32Z")
The title _Journal of **Natural** Pharmaceuticals_ makes me suspicious and I wonder how credible the magazine is. Let’s see if the original blogger can get a similar paper past _Nature_. If not, calling all peer-review journals useless based on the blog report is like throwing out all newspapers because of an article on three-headed babies in the _National Enquirer_. ---
**Author:** ![Really\_Not\_All\_That\_Bright](https://avatars.discourse-cdn.com/v4/letter/r/e8c25b/32.png) [@Really\_Not\_All\_That\_Bright](https://boards.straightdope.com/u/Really_Not_All_That_Bright)\ **Post date:** [March 5, 2014, 6:53pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/9 "2014-03-05T18:53:33Z")
Don’t peer reviewers usually put their names on things? ---
**Author:** ![wolfpup](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/wolfpup/32/10618_2.png) [@wolfpup](https://boards.straightdope.com/u/wolfpup)\ **Post date:** [March 5, 2014, 7:24pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/10 "2014-03-05T19:24:05Z")
> [@Amateur\_Barbarian](#): > > The explosion of journals (from around one per major field in the 1950s-60s) and meteoric rise of publish-at-any-cost (used to be publish or perish; now it’s publish or never get another job in this town, kid) have diluted the nature of peer-reviewed publication, yes. > > **It’s particularly bad in the money-driven fields like pharma.** (emphasis mine) Indeed it is. When pharmaceutical companies run clinical trials, they really, really want to get positive results. In typical physical science research, null results are still results. Pharma research is a rather unique and peculiar field whose challenges have often been noted. There’s a phenomenon called the “decline effect” in which some drugs have shown an inexplicable decline in efficacy over time, which some have tried to use to challenge the methodologies used in their evaluation in the first place. And something called the Kaplan-Meier curve which is supposed to show the relative therapeutic benefits of a cancer drug over a control can suddenly reverse itself or show inconsistent results in different studies. A few years ago Harvard created a program in pharmaceutical studies called “Program in Placebo Studies and the Therapeutic Encounter”, and its director, Ted Kaptchuck, has been quoted as asking the rhetorical question, “Do you think this entire field [of pharmaceutical research] is based on a foundation of magical thinking?” And the “soft” sciences like psychology and sociology tend to be less rigorous than the physical sciences, at least IMHO. Then you have the explosion in journals that was partially driven by the Internet, where you get things like shameless pay-for-publication vanity journals. On top of that there are journals with fairly explicit political agendas – like _Energy & Environment_, a journal with a special interest in undermining the credibility of climate science (not to be confused with legitimate journals that have a similar name). And the other things you say are true, too. But in the final analysis, the question the average lay person would ask is, “can you trust science?” I’d sum it up this way. A few years ago the National Institutes of Health funded a poll of 3,247 scientists which made the news because some of them admitted “faking data”. Aside from the points above about some aspects of medical research, one might note that the percentage involved even in this much-ballyhooed study used in some anti-science pontifications was 0.3%. Perhaps one might compare that with the average mass-media editorial, Internet blog, or used car dealer to see where science rates on the scale of trustworthiness. ---
**Author:** ![Steely\_Dan\_Fan](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@Steely\_Dan\_Fan](https://boards.straightdope.com/u/Steely_Dan_Fan)\ **Post date:** [March 5, 2014, 7:47pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/11 "2014-03-05T19:47:59Z")
This is like saying that lockout/tagout of energy sources when performing maintenance on industrial equipment is flawed because it doesn’t protect against someone smashing the (usually plastic) lockbox with a sledgehammer and turning everything back on while people are still working in it–you’re holding it to a standard it was never designed to meet. Just like LOTO is meant to protect against _accidental_ re-energization rather than malice, peer review was never meant to protect against outright dishonesty of the sort Bohannon perpetuated. ---
**Author:** ![Amateur\_Barbarian](https://avatars.discourse-cdn.com/v4/letter/a/59ef9b/32.png) [@Amateur\_Barbarian](https://boards.straightdope.com/u/Amateur_Barbarian)\ **Post date:** [March 5, 2014, 7:57pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/12 "2014-03-05T19:57:34Z")
> [@Steely\_Dan\_Fan](#): > > Just like LOTO is meant to protect against _accidental_ re-energization rather than malice, peer review was never meant to protect against outright dishonesty of the sort Bohannon perpetuated. Hmm. Those who recall the era of paranormal fascination at the university and research-tank level, ca. 1974 or so, will remember the astounding results and eventual downfall of the whole mess came from the researchers being unable to conceive or believe that anyone would cheat in the testing. Google “n rays.” Google “Wakefield.” Google “cold fusion.” Google the French study a few years back that came down to one woman mis-counting the samples. Google “Piltdown Man,” fa chrissakes. You really want to maintain that peer review has no mechanism, no hurdle that tests for and validates honesty in the submissions? Let’s say you’re correct: what in the fuck will it take for a little fundamental skepticism to get in the way of bogus submissions? ---
**Author:** ![Voyager](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/voyager/32/133_2.png) [@Voyager](https://boards.straightdope.com/u/Voyager)\ **Post date:** [March 5, 2014, 8:34pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/13 "2014-03-05T20:34:22Z")
This is yet another example of a basic misunderstanding how science actually works. If a dozen clowns start a journal, and submit stuff to it, and send it to each other to review, it will be peer-reviewed - but the peers will be clowns. This might a step up for some “journals.” People working in a field know the hierarchy of journals. For anyone in this field or looking for tenure one article in Science or Nature is worth a gross of papers in the Bozo Bulletin or whatever the journal mentioned in the OP is called. If a junk paper is accepted by a junk journal, there is a problem. If a junk paper is accepted by a junk journal - well, no surprise. It used to be that you needed enough money to publish something. Now if you have an axe to grind about an area where you think the mean old scientific establishment is refusing your brilliant work on squaring the circle, you can start an open access journal for almost nothing and “publish” away. Not all open access journals are bad - but you can’t treat all journals the same way. Seeing a travesty on YouTube doesn’t mean you get to say that Hollywood is producing any old crap these days. ---
**Author:** ![Voyager](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/voyager/32/133_2.png) [@Voyager](https://boards.straightdope.com/u/Voyager)\ **Post date:** [March 5, 2014, 8:38pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/14 "2014-03-05T20:38:13Z")
> [@wolfpup](#): > > And the “soft” sciences like psychology and sociology tend to be less rigorous than the physical sciences, at least IMHO. Depends on the kind of psychology you do. The kind my daughter does is mathematically driven. Statistical validity of experimental results is extremely important. Not all psychology is like that. ---
**Author:** ![Thudlow\_Boink](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/thudlow_boink/32/320_2.png) [@Thudlow\_Boink](https://boards.straightdope.com/u/Thudlow_Boink)\ **Post date:** [March 5, 2014, 9:14pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/15 "2014-03-05T21:14:19Z")
> [@Budget\_Player\_Cadet](#): > > One of my more “intellectually challenged” hippie friends posited that because of this, peer review was worthless and that we should ignore it. Say whaaaaat? It’s pretty obvious to me that the author of that linked article is taking aim at open-access journals that have lax publication standards, not against the process of peer review. ---
**Author:** ![Hentor\_the\_Barbarian](https://avatars.discourse-cdn.com/v4/letter/h/8edcca/32.png) [@Hentor\_the\_Barbarian](https://boards.straightdope.com/u/Hentor_the_Barbarian)\ **Post date:** [March 5, 2014, 9:41pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/16 "2014-03-05T21:41:06Z")
> [@Thudlow\_Boink](#): > > Say whaaaaat? > It’s pretty obvious to me that the author of that linked article is taking aim at open-access journals that have lax publication standards, not against the process of peer review. Seconded. This is about open access journals, which are shit. It isn’t an indictment of the peer review process. ---
**Author:** ![wolfpup](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/wolfpup/32/10618_2.png) [@wolfpup](https://boards.straightdope.com/u/wolfpup)\ **Post date:** [March 5, 2014, 9:51pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/17 "2014-03-05T21:51:33Z")
> [@Voyager](#): > > Depends on the kind of psychology you do. The kind my daughter does is mathematically driven. Statistical validity of experimental results is extremely important. Not all psychology is like that. Quite true, and this is a problem with making broad generalizations. I still maintain it’s true on average when compared with the physical sciences, but certainly not true in every case. I’ve known a number of research psychologists (still do) and had a great deal of respect for some but certainly not all of them; some were certainly doing valuable science, and a few had valuable interdisciplinary affiliations like cognitive science and AI. That was really a rather incidental sidelight to my post. ---
**Author:** ![Steely\_Dan\_Fan](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@Steely\_Dan\_Fan](https://boards.straightdope.com/u/Steely_Dan_Fan)\ **Post date:** [March 5, 2014, 9:52pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/18 "2014-03-05T21:52:36Z")
> [@wolfpup](#): > > And the “soft” sciences like psychology and sociology tend to be less rigorous than the physical sciences, at least IMHO. Not really. It’s just as rigorous; the rigor simply takes a different form, one that’s more tailored to the specific problems that arise with high-level research in that particular field. This is the case with other fields as well. ---
**Author:** ![Zero-syde](https://avatars.discourse-cdn.com/v4/letter/z/a3d4f5/32.png) [@Zero-syde](https://boards.straightdope.com/u/Zero-syde)\ **Post date:** [March 5, 2014, 11:04pm UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/19 "2014-03-05T23:04:19Z")
As one who spent a lot of time in the lab, the title of the argument has me practically frothing at the mouth. But, at least the OP has some merit. Is Peer Review Valid? Yes. Are all implementations of a valid system inherently valid in the same regard? Yes. Is the journal mentioned in the OP _actually implementing_ the system described? No. \<-- and that’s the problem. “Peer-review” doesn’t mean rubber stamping, or submitting a paper to a group limited to those that _agrees_ with you. It means submitting a paper to a diverse group of specialists in the field, some of whom agree and some of whom dispute your claims, for rigorous analysis. So yes, peer-review is valid, but that doesn’t mean every journal that claims to implement it does. There are _plenty_ of Christian science magazines that _claim_ peer review, right alongside their latest “evidence” of “the missing link” and “irreducable complexity.” ---
**Author:** ![Hentor\_the\_Barbarian](https://avatars.discourse-cdn.com/v4/letter/h/8edcca/32.png) [@Hentor\_the\_Barbarian](https://boards.straightdope.com/u/Hentor_the_Barbarian)\ **Post date:** [March 6, 2014, 12:32am UTC](https://boards.straightdope.com/t/is-peer-review-valid/682928/20 "2014-03-06T00:32:16Z")
> [@Steely\_Dan\_Fan](#): > > Not really. > > It’s just as rigorous; the rigor simply takes a different form, one that’s more tailored to the specific problems that arise with high-level research in that particular field. > > This is the case with other fields as well. I’m not sure what you mean by the rigor taking a different form. In any event, psychological research, at least as published in established and worthwhile journals, is certainly rigorous. In fact, psychological research often involves very sophisticated statistical techniques to model within and between individual factors to explain outcomes. [Next page](https://boards.straightdope.com/t/is-peer-review-valid/682928.md?page=2)