Politics and peer review. What comes next?

Great Debates
1

If you follow any political argument you will, sooner rather than later in most cases, come across someone stating that a certain thing is a fact due to some peer reviewed paper. Invariably, the person who questions the peer-reviewed science is labeled all sorts of things though the most used these days seems to be ‘denier’.

However, a bunch of recent studies indicate that peer review, at least in its current form, has some serious issues.

For example, Nature published a story in August of last year about a study that found over 50% of peer reviewed psychology papers could not be replicated. In fact, in that study only 39 out of 100 could be verified. Linky

Statistically it is even worse. In those original papers, 97% of the studies found a significant effect. The replications of those studies only found a significant effect 36% of the time. (Note, next time you see someone cite a peer reviewed psychology paper to strengthen their argument note that is more likely than not that the paper cannot be replicated and is therefore speculation, not science)

Now the above study was just for psychology. However it is becoming clear that this problem affects science in general and not just psychology. Besides outright fraud (which, incidentally, I believe makes up a small percentage of the problem), there are tons of badly designed experiments and just downright sloppy science. There is also the issue of Pal Review, where a small group of scientists review either their own papers or review their pals papers in a circular fashion. See this.

This is happening with medical papersand even the most prestigious publishers have problems. For example, the IEEE, those folks who set standards for electrical engineering (including the protocol that is delivering this web page to you with the 802 family of protocols) got busted with a ton of fake papers. Linky

The problems with peer review seem, from the outside, to be threefold. First, it appears that some reviewers do not dig into papers as deeply as needed because, they believe some other people will look at it and catch any problems… Second, there seem to be political considerations in that some scientists kill studies they don’t like because the studies disagree with their theories. Scientists have threatened to boycott journals that publish results that the scientists do not like. Third, the career path for scientist these days is largely (or so it appears from the outside) based upon the number of peer reviewed articles a person has published. The last puts pressure on scientists to publish as much as possible which leads to errors and fraud, plus it overloads the reviewers with papers to read. Oh, I almost forgot, journals like publishing ‘sexy’ research which also affects the peer review process.

In any case, it is apparent that peer review across the board needs to be changed.

The question is what should be done? Science has, regrettably, been politicized and it seems that everyone can find some ‘peer reviewed’ paper to fit whatever particular view point they happen to espouse.

Any ideas? How can this be fixed?

Slee

P.S. There are a ton more cites if anyone wants them. Also, retractionwatch.com is an awesome resource.

2

Something tells me that the conclusions made by some of the articles cited are, let us say, also sexed up. :slight_smile:

But one of them points to what I do remember finding before about that issue:

That is one thing that I can see has been dropped in many conspiracy minded sites that attract a lot of pseudo science. I have to mention that because I do see a lot of exaggerations coming from sites like those that need to undermine the trust that many people do have, about what many scientists do agree in many fields. And they are misleading a lot of people as I can see.

Of course one thing that it should not be missed is that the scandal described concentrated mostly on the “open access” journals that indeed are being criticized by the very same people that do want to make science better. And that BTW is one of the solutions, one just has to be vigilant but also one has to be careful to not fall for the other extreme.

This criticism is valid for a good number of open access journals and some shady ones but it should not be assigned to all of science, for the reason that while the number of bad papers mentioned is a bad one, has to take into account how small really is with the total numbers of publications made in the scientific world.

http://blogs.nature.com/news/2014/05/global-scientific-output-doubles-every-nine-years.html

So that is just to put the main point in the OP in perspective. As for remedies on how to identify the bad, the articles cited in the OP are part of the answer, and then another is to, well do more science. As 538 puts it in their “Science isn’t broken” article:

3

Got any examples? (I.e., any examples of scientists threatening to boycott journals for political reasons rather than from encountering bullshit and refusing to support a journal that would publish it?)

4

I agree that some published papers are of low quality and that the buzz-phrase ‘peer-reviewed’ is overused and used ignorantly.

However, I’m not aware that this has much effect on politics. AFAIK, the most outlandish political claims about economics, evolution, environment, etc. do NOT come from anything resembling academic papers, let alone peer-reviewed papers. OP, do you have examples of bad ‘peer-reviewed’ papers with a significant effect on political debate?

5

This is a good lesson in why people need to learn how to read peer-reviewed papers, particularly to determine how solid the evidence is and judge whether the employed methodology is credible. Unfortunately, I think a lot of people tend to skip to the conclusions and forget about the rest of it. As far as psychology goes, those are often inherently iffy due to small sample sizes or selection bias. When these folks study tender mental issues, it’s sometimes hard to get people into these studies.

On the broader topic of peer-review, I’ve spoken with others about this in the past. My info is purely anecdotal but I’ve been told the folks conducting peer-review are often not experts in the academic domain within which the papers are written. Hence, some of these problems may not be due to a lack of enthusiasm. It may just be they simply lack expertise on these topics. I would also note there are a lot of journals out there and not all of them are objective. Some publish papers because they have an axe to grind.

6

Gigo, did you bother to read any of my cites? I suspect not and you went with a knee jerk defense without actually understanding the people involved.

The Reproducibility Project specifically targeted highly respected journals, had the replicators work with the original scientists and tested easily reproducible experiments. Yet only 36% of the replicated studies showed significant results. Additonally, the results were published in Nature. Is Nature now a haven for conspiracy theorists? Is that your position?

Additionally, the problem is deeper than just finding bad papers. The bad papers are published, taken as true by politicians and scientists. The pols use them to push certain policies and the scientists take the results as true and base further research on faulty papers.

And, Gigo, yes there are tons of papers published. The Reproducibility Project specifically targeted easy to replicate studies and most could not be reproduced. The logical conclusion is that studies that are harder to reproduce will have lower reproducibility rates. In other words, it is the reproducibility failure ratio that matters and a 60% failure rate is horrid. Science isn’t broken but peer review sure as hell is.

tomndeb, I will hunt up a cite for the boycot issue. Sadly a kerfluffle over Elsevier has jacked up the google results and I can’t find what I am looking for right now.

Slee

7

Wrong as my quote from one of them showed. Indeed the issue was with most of the new open access journals, that curiously are preferred by many pseudoscientist groups out there.

No because in your line about “This is happening with medical papers and even the most prestigious publishers have problems.” the link you made there pointed to the link I made about the open access journals being busted. And the writers do mention that:

The issue you were mentioning that was not quite really aplicable to all fields of science was specifically in the psychology field. Just remembering how Freud has not been taken seriously nowadays and other issues I already was aware about how unreliable that field is. It is however not good to then conclude that other fields are just the same.

The problem with that is that there is a lot of bias in that conclusion of yours, it’s mostly your opinion. And as pointed before you actually cited the solution to the issue. More research is needed to be done to overturn the bad papers. And that is usually what has happened in science before.

I think this is missing the point, peer review is not the only tool that science uses, but then indeed reproducibility and new advances in technology or data confirm or overturn past papers. As your OP deals with politics then the most important thing that is needed here is to point at the issues where you think the science is wrong and the politicians are following what most scientists (that are wrong) are recommending, what could that be?

8

[QUOTE=GIGObuster]
No, because in your line about “This is happening with medical papers and even the most prestigious publishers have problems.” the link you made there pointed to the link I made about the open access journals being busted. And the writers do mention that:

[/QUOTE]

Just to expand a bit more in this item, indeed that link was not about “prestigious publishers have problems [too]” but about open access ones that have taken a drubbing lately, and not just by investigations like that one you linked at but also by serious scientists that are seeing that many pseudo scientists are reaching for journals like that to make their misleading papers to be just more than just their musings made in the court of public opinion. One just needs to look at the latest 9/11 thread on how the “Architects and Engineers for 9/11 Truth” found it easy to twist those open access system journals.

http://boards.straightdope.com/sdmb/showthread.php?t=800493&page=3

9

I’m not actually all that surprised about the lack of reproduciblility. Before reading this I would guess that around 50% of studies are false positives. If you then say that for those that are real, the repeat study had only about an 80% power to detect the result, you end up at around the number shown.

But for the most part, scientists don’t rely on a single study when making decisions. The first study showing an effect is always viewed as an interesting potential result, but rarely as definitive truth. Over time those results that fail to reproduce will be forgotten, and only those for which additional confirmatory evidence is found in future studies will be taken as part of the bedrock of the field.

The real problem is the media which will take a single study, misrepresent the findings and write glowing headlines about how eating blueberries will cure your cancer.

10

This is a problem, but it’s a separate problem than the one this thread identifies.

Even ignoring open access journals, there is still a problem with replicability.

11

You clearly have no clue as to how the IEEE works, and neither does the person who wrote the article in your link. I do, having been general chair and program chair of several IEEE conferences, and being on the steering committee of one for a long time.

First, getting a conference authorized is mostly about showing it will make or at least not lose money. There are thousands upon thousands of IEEE conferences and workshops, and the people in IEEE HQ have no way of evaluating the quality of the program committee, if any. In fact the peer review process used, if any, in not part of the application form.
Then, the process of submitting papers from a conference (which I’ve done quite a lot) consists of generating a packing list with all the components of the publication, and sending pdfs of the papers. No one reads them - of course - and the IEEE staff wouldn’t understand them if they did read them. And being in the repository is no evidence of quality or even accuracy. My joke column about base one arithmetic is in there. You are not the only one making this mistake - I frequently get emails inviting me to submit a paper - as an expert - on a subject I write a humor column about.

Oh, but that proves it is a mess, you say. But that would show a lack of understanding of what counts. First, it is the prestige of the journal or conference - ones in China like the ones the made up papers got into don’t have much at all, neither do the for profit conferences we all get spammed about. Second is the citation count of the paper. A write-only journal doesn’t have much of an impact for good or bad. I bet that none of the made-up papers ever got cited. In fact I bet none of them ever got actually reviewed. My guess is that the people setting up the conference did so for their professional advancement, got almost no papers submitted, and thus took anything that came in without question. So these papers probably weren’t even peer reviewed, and thus hardly count against it.
I’ve read thousands of reviews, and they are of varying quality. Some just say good paper, some don’t get the point of the paper at all, some come from an opposed viewpoint and aren’t fair, and some are insightful. A good editor or program chair weighs them, where they come from, and the habits of the reviewer.) We keep the average score for each reviewer, since a 7 out of ten for someone who usually gives 4s is better then from someone who usually gives 9s.
And of course editors are often hard pressed to find reviewers, and reviewers may be too busy to really research a paper about something they aren’t up on.

But in general peer reviewed is better than not. But no working scientist would call it a guarantee of quality.

12

If there was never a problem with replicability, no one would ever need to replicate. You seem to be missing the entire point of publishing in enough detail to make an attempt at replication possible. Scientists make mistakes, scientists get over-enthusiastic, with cold fusion being the textbook case.
I read something that said the attempts at replication of the psychology papers were done in different cultures from the original, which throws in a new variable, but I haven’t studied it in depth to be sure.
Hell, I new someone doing a PhD in Physics who discovered that a widely accepted result in his area was wrong. Bummer - and not worth a PhD, alas, so he had to work around it. This stuff happens.

13

It’s not a question of never being a problem. It’s what rate of non-replicability is acceptable? If 5% is not replicable, that may be fine. If 95% is not replicable, that’s worse. Scientists make mistakes, but if a system yields a rate of mistake high enough to call into doubt results in general, then the system needs to be re-examined.

14

The actual rate of non-replication is probably horrible - but most papers are not important enough for anyone to care about. Reviewers are obviously not going to replicate anything.
In my field, at least, most contradictions of published work come in the Previous Work section of a paper, and maybe in the results where new results are compared to previous results.
My daughter has a PhD in psychology, and in more than one case really nice results didn’t look so nice when the experiment was rerun. That stuff doesn’t get published, but if someone gets lucky twice in a row it will. Psychology is not physics, and even physics isn’t perfect.

15

Peer review has nothing to do with replicability though. Reviewers are not expected to attempt to reproduce the results, they’re checking for completeness, relevance, coherency, stuff like that.

16

I’ve published stuff that turned out to be incorrect, subsequently corrected by me or others. I don’t really see what the big deal; that’s how science works. Obviously outright fraud is bad, but that’s a separate issue. I’m not seeing the crisis the OP is grasping at.

17

Post snipped.

So, in other words, any conference paper that is published by the IEEE is of unknown quality and could be bullshit. And that the people who accept the bullshit papers do so for career advancement. And the IEEE is ok with that. Good to know. (For kicks, I went to the list of pulled papers from the IEEE and started google searching randomly. The 5th paper I searched has been cited. The paper is “Using knowledge for data mining of software processes in knowledge based LMS” cited by “A generalizable knowledge framework for semantic indoor mapping based on Markov logic networks and data driven MCMC”. So yes, the bogus papers are being cited)

Regarding the last sentence of your reply, scientists may understand peer review isn’t a guarantee of quality, but politicians, talking heads and a bunch of other people do.

As far as the papers that the Reproducibility Project tried to reproduce, you can go look. They list the paper, what they did to reproduce the paper, the results and anything they know of that might cause the reproduction to fail.

Gigo, the reproducibility problem is everywhere. Medical studies have serious problems. According to Nature:

Link.

Preclinical research has the problem.

and more drug research

And cancer research.

Link.

Where politics comes in is that these studies get picked up and acted upon because ta da they were peer reviewed. Advice is given, for example the nutrition guidelines in the U.S. that were based on the Seven Countries Study. The seven countries study supposedly showed that saturated fats led to high cholesterol which led to heart attacks. That caused the food industry to stop using saturated fats and instead use carbohydrates instead while the government issued the ‘Recommended dietary goals for the American People’. That recommendation was to increase carb intake to ~60% of calories and to reduce fat intake to <30% of calories. So fats became the boogyman, Yay carbs! And everyone got fat and heart attacks stayed roughly the same. Hereis a decent history of the whole debacle.

Peer review isn’t good enough. We need a better system.

Slee

18

Yeah, this, and it applies pretty much to the entirety of the OP.

There is nothing wrong with a rational analysis of peer review inadequacies, with pointing out that there are less than stellar journals, or with other informed efforts to improve the integrity of the scientific process. But the OP is doing no such thing. The OP is piling on, in an outrageously uninformed and partisan manner, to the anti-science plague that already afflicts America, which has been well documented in the literature and well described in the October 29 - November 4 issue of New Scientist. The theme of the issue was “Unscientific America: A dangerous retreat from reason” and documents with depressing clarity how the politicization of scientific ignorance is driving US public policy farther and farther from rational decision-making.

The OP stated the following right at the start:
If you follow any political argument you will, sooner rather than later in most cases, come across someone stating that a certain thing is a fact due to some peer reviewed paper. Invariably, the person who questions the peer-reviewed science is labeled all sorts of things though the most used these days seems to be ‘denier’.
To me this makes the intent abundantly clear. This is not an attempt to “improve” science, it’s a flagrant attempt to discredit it. And in support of this thesis, it offers deeply flawed cherry-picked examples of papers in psychology, which is among the least rigorous of all the soft and non-rigorous social sciences, and medical research, which is a very productive yet tremendously complex field especially in notoriously difficult areas like testing cancer drug efficacy in clinical trials. Apparently we’re supposed to conclude that favored right-wing policies on issues like climate change should prevail on the basis that science is so terribly flawed that you may as well just ignore it and go with your gut. And this is just sad, abject nonsense and exemplifies exactly the dangerous problem that the New Scientist was lamenting.

19

Peer review is a go/no-go gauge. If a wild paper is published through a vanity press with no peer review, that is generally enough to disqualify it as a reference in public debate. So, when various Creationist, Truther, Flat-Earther, Race-Truth, and Economic Voodoo books (e.g., The Bell Curve), are waved around to influence policy, pointing to the lack of peer review is a legitimate response.
The saturated fat problem did not arise through a failure of peer review. It is possible, but very unlikely, that any reviewer would have caught the cherry-picking in which Keys engaged. That is not how peer review works.

Beyond that, peer review does nothing to make the presentation in a book accurate, (and I know of no one in any science or technical field who would claim it does).
You appear to be searching for an answer when there is not a problem. I certainly have never heard Trump, (or Cruz or Bush or even Carson–who may not know what peer review is), try to promote or dismiss a work based on it being peer reviewed. That is much more likely a response for a message board trying to weed out crackpots than it is a political issue.

Would it be nice to have a review process that was better than the current situation? I guess so. But short of going to a 1930’s Soviet Union style publishing system, that is not going to ever happen. There are hundreds of thousands of scientists, engineers, and other researchers, (not even getting into Liberal Arts), who are trying to publish findings on a daily basis. There are simply never going to be enough people standing at the gates to monitor every article in every field before it can be sent to the printer.
We do not want a Soviet style gate system, (which limited output and did rely on politics), so we are going to have to muddle through on what we have.

Is there a specific, (or even general), proposal that you could envision? How would it work? Where do you find the huge number of people with the time and training to actually review the texts and recognize (for example) that people interviewed in near-famine post-war conditions during Lent were not an appropriate population? My guess will be that the best effort might be some sort of Artificial Intelligence based software, getting humans out of the loop, but I suspect that even with such software, humans with their psychological biases and similar problems will still find ways to get past AI–and without even intending malice.

20

Damn, I forgot to mention the year when referencing the October 29 - November 4 issue of New Scientist. It was in 2011. Also that access requires a subscription so I can’t link to the articles. But here’s the start of one of the articles that reinforces my point, the one titled “Science in America: Decline and fall”, the one that came with the preamble “The US is the most powerful scientific nation on Earth and yet the status of science in public life has never been so low”:
“The big thing we are working on now is the global warming hoax. It’s all voodoo, nonsense, hokum, a hoax.” So said Michele Bachmann, a candidate for the Republican nomination for president, in 2008. Bachmann also thinks that the human papillomavirus (HPV) vaccine can cause mental retardation and that science classes should include creationism. “What I support is putting all science on the table and then letting students decide. I don’t think it’s a good idea for government to come down on one side of a scientific issue or another, when there is reasonable doubt on both sides.”

Bachmann’s rival, Texas governor Rick Perry, advocates biblically based abstinence-only sex education. He argues that evolution is “a theory that is out there - and it’s got some gaps in it”. On climate change, Perry says “the science is not settled… just because you have a group of scientists that have stood up and said here is the fact… Galileo got outvoted for a spell”.

Former House Speaker Newt Gingrich tells voters that embryonic stem cell research is “killing children in order to have research materials”. Rising Republican star Herman Cain claims there is no scientific evidence that homosexuality is anything other than a personal choice. Republicans diverge from anti-science politics at their peril. When leading candidate Mitt Romney said: “I believe based on what I read that the world is getting warmer… humans contribute to that”, conservative radio commentator Rush Limbaugh responded with “Bye bye, nomination”. Romney back-pedalled, saying, “I don’t know if it’s mostly caused by humans.”