So this story came to my attention a while back:
http://www.sciencemag.org/content/342/6154/60.full
One of my more “intellectually challenged” hippie friends posited that because of this, peer review was worthless and that we should ignore it. I think this is a load of crap. Peer-review is a necessary first step; all this does is show that many journals are not exactly scrupulous with it. What do you guys think?
The explosion of journals (from around one per major field in the 1950s-60s) and meteoric rise of publish-at-any-cost (used to be publish or perish; now it’s publish or never get another job in this town, kid) have diluted the nature of peer-reviewed publication, yes. It’s particularly bad in the money-driven fields like pharma.
All fields would be interested to hear of an alternative system. Got one?
(Me, I’d like to see a lower bar established, to eliminate the practice of ten minor and nearly meaningless publications in place of one or two that actually advance knowledge and understanding in the field. But nobody listen to Zathras.)
How do the older established peer-reviewed journals stack up against the new ones when it comes to reliability? I noticed that the Journal of Natural Pharmaceuticals is less than 4 years old.
There’s a difference between “peer review” as it is practiced by publishers, and the fact that scientists the world over spend a lot of time replicating important experiments and verifying contending theories.
Any particular paper has a good chance of being bullshit. But in ten or twenty years, when that paper’s main idea has been fleshed out, tested repeatedly, and come to be the consensus view, you can bet that it’s very likely correct. Or as close as you get in science.
But to a certain extent, whenever you see a news story that begins with “Studies show…”, there’s a good chance it’s crap.
Its website certainly hasn’t been reviewed (by peers or otherwise):
This is all true, and is important to keep in mind when reading news of scientific research. The fact that something appeared in a peer-reviewed journal doesn’t in any way make it correct.
But, I think this is also a little beside the point. The point of peer review isn’t to validate the research is correct. To do that, the reviewer would have to actually try to reproduce the results of the experiment. The point of peer review is, roughly speaking, to validate the quality of the paper being submitted. Are the statements contained within it coherent and expressed in a comprehensible way? Is it free of obvious flaws and errors? Is the research significantly noteworthy to be worthy of inclusion in this journal? That sort of thing.
If a paper is passes peer-review, and its conclusions are later proven wrong by subsequent research, that’s no failing of the review system. But if a paper which is nonsense, or obviously fraudulent, somehow makes it into a peer-reviewed journal, then that absolutely is a failing of the peer-review system, and suggests that the reviewers were rubber stamping things without bothering to read or understand them. (Edited to add: Or it suggests that the publisher is lying when claiming they had the paper reviewed by experts in the field.)
Regarding the question in the subject, some form of quality control by academic journals seems like a must. I think probably peer-review is still the best method of quality control, but no doubt it isn’t perfect and there is room for improvement.
Doing away with peer review without a suitable replacement doesn’t make much sense…
The title Journal of Natural Pharmaceuticals makes me suspicious and I wonder how credible the magazine is. Let’s see if the original blogger can get a similar paper past Nature.
If not, calling all peer-review journals useless based on the blog report is like throwing out all newspapers because of an article on three-headed babies in the National Enquirer.
Don’t peer reviewers usually put their names on things?
(emphasis mine)
Indeed it is. When pharmaceutical companies run clinical trials, they really, really want to get positive results. In typical physical science research, null results are still results.
Pharma research is a rather unique and peculiar field whose challenges have often been noted. There’s a phenomenon called the “decline effect” in which some drugs have shown an inexplicable decline in efficacy over time, which some have tried to use to challenge the methodologies used in their evaluation in the first place. And something called the Kaplan-Meier curve which is supposed to show the relative therapeutic benefits of a cancer drug over a control can suddenly reverse itself or show inconsistent results in different studies. A few years ago Harvard created a program in pharmaceutical studies called “Program in Placebo Studies and the Therapeutic Encounter”, and its director, Ted Kaptchuck, has been quoted as asking the rhetorical question, “Do you think this entire field [of pharmaceutical research] is based on a foundation of magical thinking?”
And the “soft” sciences like psychology and sociology tend to be less rigorous than the physical sciences, at least IMHO. Then you have the explosion in journals that was partially driven by the Internet, where you get things like shameless pay-for-publication vanity journals. On top of that there are journals with fairly explicit political agendas – like Energy & Environment, a journal with a special interest in undermining the credibility of climate science (not to be confused with legitimate journals that have a similar name).
And the other things you say are true, too.
But in the final analysis, the question the average lay person would ask is, “can you trust science?” I’d sum it up this way. A few years ago the National Institutes of Health funded a poll of 3,247 scientists which made the news because some of them admitted “faking data”. Aside from the points above about some aspects of medical research, one might note that the percentage involved even in this much-ballyhooed study used in some anti-science pontifications was 0.3%. Perhaps one might compare that with the average mass-media editorial, Internet blog, or used car dealer to see where science rates on the scale of trustworthiness.
This is like saying that lockout/tagout of energy sources when performing maintenance on industrial equipment is flawed because it doesn’t protect against someone smashing the (usually plastic) lockbox with a sledgehammer and turning everything back on while people are still working in it–you’re holding it to a standard it was never designed to meet.
Just like LOTO is meant to protect against accidental re-energization rather than malice, peer review was never meant to protect against outright dishonesty of the sort Bohannon perpetuated.
Hmm.
Those who recall the era of paranormal fascination at the university and research-tank level, ca. 1974 or so, will remember the astounding results and eventual downfall of the whole mess came from the researchers being unable to conceive or believe that anyone would cheat in the testing.
Google “n rays.” Google “Wakefield.” Google “cold fusion.” Google the French study a few years back that came down to one woman mis-counting the samples. Google “Piltdown Man,” fa chrissakes.
You really want to maintain that peer review has no mechanism, no hurdle that tests for and validates honesty in the submissions? Let’s say you’re correct: what in the fuck will it take for a little fundamental skepticism to get in the way of bogus submissions?
This is yet another example of a basic misunderstanding how science actually works.
If a dozen clowns start a journal, and submit stuff to it, and send it to each other to review, it will be peer-reviewed - but the peers will be clowns.
This might a step up for some “journals.”
People working in a field know the hierarchy of journals. For anyone in this field or looking for tenure one article in Science or Nature is worth a gross of papers in the Bozo Bulletin or whatever the journal mentioned in the OP is called. If a junk paper is accepted by a junk journal, there is a problem. If a junk paper is accepted by a junk journal - well, no surprise.
It used to be that you needed enough money to publish something. Now if you have an axe to grind about an area where you think the mean old scientific establishment is refusing your brilliant work on squaring the circle, you can start an open access journal for almost nothing and “publish” away.
Not all open access journals are bad - but you can’t treat all journals the same way.
Seeing a travesty on YouTube doesn’t mean you get to say that Hollywood is producing any old crap these days.
Depends on the kind of psychology you do. The kind my daughter does is mathematically driven. Statistical validity of experimental results is extremely important. Not all psychology is like that.
Say whaaaaat?
It’s pretty obvious to me that the author of that linked article is taking aim at open-access journals that have lax publication standards, not against the process of peer review.
Seconded. This is about open access journals, which are shit. It isn’t an indictment of the peer review process.
Quite true, and this is a problem with making broad generalizations. I still maintain it’s true on average when compared with the physical sciences, but certainly not true in every case. I’ve known a number of research psychologists (still do) and had a great deal of respect for some but certainly not all of them; some were certainly doing valuable science, and a few had valuable interdisciplinary affiliations like cognitive science and AI. That was really a rather incidental sidelight to my post.
Not really.
It’s just as rigorous; the rigor simply takes a different form, one that’s more tailored to the specific problems that arise with high-level research in that particular field.
This is the case with other fields as well.
As one who spent a lot of time in the lab, the title of the argument has me practically frothing at the mouth. But, at least the OP has some merit.
Is Peer Review Valid? Yes.
Are all implementations of a valid system inherently valid in the same regard? Yes.
Is the journal mentioned in the OP actually implementing the system described? No. <-- and that’s the problem.
“Peer-review” doesn’t mean rubber stamping, or submitting a paper to a group limited to those that agrees with you. It means submitting a paper to a diverse group of specialists in the field, some of whom agree and some of whom dispute your claims, for rigorous analysis.
So yes, peer-review is valid, but that doesn’t mean every journal that claims to implement it does. There are plenty of Christian science magazines that claim peer review, right alongside their latest “evidence” of “the missing link” and “irreducable complexity.”
I’m not sure what you mean by the rigor taking a different form.
In any event, psychological research, at least as published in established and worthwhile journals, is certainly rigorous. In fact, psychological research often involves very sophisticated statistical techniques to model within and between individual factors to explain outcomes.