Reproducibility Crisis

In My Humble Opinion
1

Dozens of major cancer studies can’t be replicated | Science News

After eight years, a project that tried to reproduce the results of key cancer biology studies has finally concluded. And its findings suggest that like research in the social sciences, cancer research has a replication problem.

Researchers with the Reproducibility Project: Cancer Biology aimed to replicate 193 experiments from 53 top cancer papers published from 2010 to 2012. But only a quarter of those experiments were able to be reproduced, the team reports in two papers published December 7 in eLife .

This is a phneomenon which has already attracted a lot of attention, particularly in the field of psychology, and the purpose of the new study was to put major cancer studies to the same test.

That said, I personally think that the issue is a much bigger one in the field of psychology, even if medical tests fail at the same rate. That’s because individual psychological studies have an import that goes far beyond that of medical studies.

This is partially because - at least in the case of drugs - there are FDA regulations that would impose a rigor that goes beyond individual studies, while there are no comparable constraints on psychological treatments.

But more importantly, because the impact of medical studies is mostly on doctors, who are professionals and know to assess the overall weight of evidence. But psychological concepts have a direct impact on broader society, and to the extent that incorrect notions get widespread acceptance based on invalid studies, it can be seriously harmful. (Nutrition would be more similar to psychology than medicine.)

ETA: that said, the fact that such a high percentage of studies fail to be reproduced suggests something seriously flawed in the process.

2

I’d say more to do with perverse incentives.

“Publish or perish” is for real. The true goal is to advance knowledge and improve health. But the way we (hospitals, universities, whoever) evaluate success is by number of publications.

It’s like the testing paradigm in US public schools. The goal isn’t high test scores - the goal is to produce educated, functional members of society and standardized tests are merely the method we have chosen to evaluate success.

The ‘win’ condition is clear - steer effort towards the evaluation metrics since that is how one will be judged. Unfortunately, the metrics we employ are effectively divorced from the goals we want to achieve.