It is a fairly well established fact in medical research that a large proportion (possibly significantly more than 50%) of published trial results are wrong, in the sense that they confirm a relationship which later trials are unable to replicate.
I am interested in locating research that gives similar information for other fields such as physics. I guess the specific questions are:
How many experimental results are reported that can’t be replicated?
How many theories or models are reported that are invalidated by observations made BEFORE the theory is published?
How many theories or models are reported that are falsified by later observations?
(I’m pretty sure the answer to the last question is “almost all of them, eventually”, so don’t bother answering that one unless you have a good cite supporting it or evidence to the contrary)
What do you cite for that initial claim on which you base your request?
I wouldn’t be surprised if it’s true the way that kind of research is done with sample sizes too small to be relevant, raw supposition serving as hypothesis, aggragate research studies and the like. But do you have a source to support that, other than one like that provided in the previous post which uses the same kind of faulty methodology that it is criticizing?
I used a prejudicial description of the cited report there. It’s methodology may not be faulty, rather it decribes a ‘means’ used by faulty researchers rather than directly addressing the ‘why’ expressed in it’s title.
I believe one standard deviation is the threshold for assuming a result to be reasonably certain not to be due to chance. That means that a properly designed and executed study has about a one-in-twenty chance of being wrong.
This is heavily dependent upon the field you are talking about. For example, I imagine the percentage of wrongness in mathematical papers is remarkable small whereas in psychology the results are a lot more subject to opinion and therefore likely have a much higher percentage of “wrongness”.
I work in engineering where I would say we fall in the middle. I think extremely little of the peer-reviewed research is actually wrong, though there is likely a reasonable amount that look at only a small subset of the overall problem that might give and incorrect (but reproducible) answers.
However, wrong doesn’t mean bad. I have papers where we had data that supported a particular hypothesis, and we published it as such. Later studies, using new techniques, showed that the hypothesis wasn’t correct.
The first studies were wrong, but they weren’t invalid. The conclusions were our best explanation based on the data we had at the time. Over time, our hypotheses evolved based on new observations.
This is how science works. Any true scientist isn’t afraid of being wrong. I’m usually wrong. I revel in being wrong. Being wrong means something interesting is happening. Some of our most interesting results have come about because I was wrong.
If you are right all the time, you are not working on interesting or important topics!
I think you’re thinking of 2 standard deviations, which cover roughly 95% of the variation (2-tailed), leaving 5% left over, while 1 SD is 68% and 32%. However, this assumes perfectly normal data, and fewer subjects will require larger differences to confidently conclude a probably difference.
To be really nitpicky, alpha of 0.05 says that you concluded that there was an effect when there wasn’t in reality; this isn’t only way you can be wrong. You can conclude no difference when one in fact exists, while you can also reach the right conclusion but make the wrong assumptions/methods.
Well, what do you mean by wrong? According to the OP, you are only wrong in the sense of number three, which most stuff is. I get the feeling they are looking more at numbers 1 and 2, and thus you were only wrong if your data wasn’t reproducible.
It’s also worth noting that just because something in unreproducible, that doesn’t mean it’s “wrong”. That is, it’s perfectly possible to do a study completely correctly and still be unreproducible.
In the medical arena, there is another factor involved, money. There is a tremendous amount of grant money available for medical research, and you need citations of your work and a claim of success to get more grant money. So the easy way to do that is design your research to find a pre-determined result that you can claim as ‘success’. Others are likely to cite you without scrutiny of your methods or results since they are doing the same. In the cite from njtt, about 20% of the author’s citations are his own work. Then add in aggragate research techniques which simply ‘average’ the results of other studies, often without regard to their validity, and you have created a mess. To illustrate that, imagine an engineering study that concludes a new bridge is structurally sound because of the large volume of studies showing existing bridges to be structurally sound.
Ok, firstly I have to concede that the most readily available cite is the Ioannidis paper in PLoS 2005, already cited in this thread.
The conclusion (although possibly the extent is overstated) is socially and mathematically plausible, and he isn’t the first to make it.
You can’t design a trial which doesn’t have a risk of a false positive, and there are strong pressures to publish early positives.
That wasn’t my question though, and no one has yet addressed the question.
We KNOW that science works through correction and falsification - continuous new hypotheses to account for continuing new information.
Does anyone know though, how much research outside of medicine is wrong, based either on failure to confirm (ie, the results are not correct) or failure to account for existing knowledge at the time (ie, the hypothesis doesn’t explain known facts)?
Thats is an interesting and important question. If there are no more questions, class is dismissed.
That’s a very difficult question to answer. I would say that in an active research field, clearly wrong results don’t stay wrong forever - it may take some time if the wrong result is part of the status quo, but eventually it’ll be corrected.
For other areas where there is not widespread interest, “wrong” results may hang around for a long time. If few people work on the problem, it is less likely that someone will try to reproduce it.
Science would make more progress faster (in my opinion) if more people would take the time to reproduce other’s results, rather than assuming they are correct to start with and building from there. Many students aren’t trained this way - their advisors say “don’t waste time - XX group has already shown that!” I think this leads to bad science.
I have a friend that ended dropping out of grad school because their dissertation work hinged on results published by a previous lab member. After wasting 3 years and all of their self-confidence trying to get their experiments to work (and getting told by their advisor how stupid, lazy and incompetent they were on a weekly basis) they finally figured out that all of the previous (published!) work was an artifact due to bacterial contamination.
In my opinion, the previous lab member was either extremely sloppy or deliberately falsified their results. But they had a government job in a foreign country and were now untouchable. Easier for the advisor to get rid of my friend, quietly ditch the project and never speak of it again.
The Ioannidis paper shows how the results of research in medicine could be prone to error because of misuse of statistics to qualify results. I didn’t check Ioannidis’s cites to see how he justifies his supposition that there is such a high rate of problems to begin with in the field of medicine either. Other fields may not be as susceptible to that problem because larger samples can be studied more easily. I don’t know if there are numbers available. But of you research the ‘Sokal Affair’ you might find citations that will help. The problems of non-science, mis-science, and mal-science abound.
There are certainly a lot of published studies out there that are extremely poorly designed. When I was still doing my postgrad work, my research group had a regular meeting in which we would practice sitting down with a paper that was relevant to one of our projects and systematically picking it apart, the results were somewhere between alarming and terrifying.
We were finding significant methodological problems (almost always with the controls) with about half the papers, and evidence of what had to be malfeasance or major incompetence about 10% of the time. My favourite was a study which was attempting to show no change in behaviour of a cell population after a couple of weeks of treatment, the “after” cell field picture was identical (same cells, same media artefacts) to the “before” picture, it was just inverted :eek:
You’ve also got to take into account just how ambitious the claims in a paper are. A paper might, for instance, say something like “These data are consistent with the predictions of Model A”. Which is probably true, but it doesn’t imply that Model A is correct. If someone later finds data which isn’t consistent with Model A, and everyone switches to Model B instead, but the data in the original paper is also consistent with Model B, does that make the first paper wrong? Or maybe the paper says something like “These data are consistent with either Model A or Model B, but a more sensitive experiment along the same lines may be able to distinguish between them”?
Yeah “consistent with” or even “not inconsistent with” are every scientists favourite weasel words. And it’s not just the author who will optimistically evaluate the work, I’ve seen a lot of medical and scientific product websites that give you a long list of papers to support their claims, some of which might even have a vaguely relevant subject.
In particular you need to be very cautious about anything that is linking in-vitro cell culture work with a clinical situation. My own project produced in-vitro results in the exact opposite direction we were expecting, but either sets of results could have been described in such a way as to be consistent with a positive clinical picture, I made this clear to my sponsors and clinical supervisor, a lot of people don’t.
