# Is there a name for this effect in scientific publishing? **URL:** \ **Category:** Factual Questions\ **Created:** [April 21, 2014, 8:23pm UTC](https://boards.straightdope.com/t/is-there-a-name-for-this-effect-in-scientific-publishing/686630 "2014-04-21T20:23:38Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Budget\_Player\_Cadet](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/budget_player_cadet/32/205_2.png) [@Budget\_Player\_Cadet](https://boards.straightdope.com/u/Budget_Player_Cadet)\ **Post date:** [April 21, 2014, 8:23pm UTC](https://boards.straightdope.com/t/is-there-a-name-for-this-effect-in-scientific-publishing/686630/1 "2014-04-21T20:23:38Z")
> **[Significant](https://xkcd.com/882/)** > > 'So, uh, we did the green study again and got no link. It was probably a--' 'RESEARCH CONFLICTED ON GREEN JELLY BEAN/ACNE LINK; MORE STUDY RECOMMENDED!' Essentially publishing so many randomized controlled trials that by sheer probability, something incredibly unlikely is almost certain to come out (note how even with P = 0,95 of a correct result, by the end of the 21 tests, the chances of no false positives is still below 50%). Also present in the infamous 2012 Seralini et al paper, where, with 20 groups, there was almost guaranteed to be one that was “shockingly” more cancerous (although that’s not the biggest issue with the study by a long shot). Is there a name for this? ---
**Author:** ![Musicat](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/musicat/32/20189_2.png) [@Musicat](https://boards.straightdope.com/u/Musicat)\ **Post date:** [April 21, 2014, 8:30pm UTC](https://boards.straightdope.com/t/is-there-a-name-for-this-effect-in-scientific-publishing/686630/2 "2014-04-21T20:30:05Z")
One name would be data mining. ---
**Author:** ![Colibri](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/colibri/32/1841_2.png) [@Colibri](https://boards.straightdope.com/u/Colibri)\ **Post date:** [April 21, 2014, 8:31pm UTC](https://boards.straightdope.com/t/is-there-a-name-for-this-effect-in-scientific-publishing/686630/3 "2014-04-21T20:31:47Z")
> [@Budget\_Player\_Cadet](#): > > [xkcd: Significant](http://xkcd.com/882/) > > Essentially publishing so many randomized controlled trials that by sheer probability, something incredibly unlikely is almost certain to come out (note how even with P = 0,95 of a correct result, by the end of the 21 tests, the chances of no false positives is still below 50%). Also present in the infamous 2012 Seralini et al paper, where, with 20 groups, there was almost guaranteed to be one that was “shockingly” more cancerous (although that’s not the biggest issue with the study by a long shot). Is there a name for this? It’s called a [Type I error](http://en.wikipedia.org/wiki/Type_I_and_type_II_errors) (false positive). Reviewers do consider this when there are a large number of tests being conducted. Note that a Type I error can occur even with a single test, it’s just that the more tests there are the more probable they will occur. ---
**Author:** ![njtt](https://avatars.discourse-cdn.com/v4/letter/n/ecd19e/32.png) [@njtt](https://boards.straightdope.com/u/njtt)\ **Post date:** [April 21, 2014, 8:33pm UTC](https://boards.straightdope.com/t/is-there-a-name-for-this-effect-in-scientific-publishing/686630/4 "2014-04-21T20:33:50Z")
The [file drawer effect](http://skepdic.com/filedrawer.html), perhaps. I am not sure whether you understand that point is that the green jellybean experiment is the only one that is actually going to get published, and the fault is not so much with the scientists as with the journal editors (and journal economics). ---
**Author:** ![njtt](https://avatars.discourse-cdn.com/v4/letter/n/ecd19e/32.png) [@njtt](https://boards.straightdope.com/u/njtt)\ **Post date:** [April 21, 2014, 8:44pm UTC](https://boards.straightdope.com/t/is-there-a-name-for-this-effect-in-scientific-publishing/686630/5 "2014-04-21T20:44:01Z")
_Missed edit window:_ There has been some talk in recent times, now that data can be stored, and made available online, so much more cheaply electronically than it ever could be on paper, of publishing journals devoted to negative results. Perhaps some have even started up by now. However, as the page charges for open access pay-to-publish journals show, even digital publication costs significant money (especially if, as tends to be the case with scientific research reports, you need to have figures and graphs and stuff), so I am not sure how well this can work. ---
**Author:** ![Smeghead](https://avatars.discourse-cdn.com/v4/letter/s/f1d935/32.png) [@Smeghead](https://boards.straightdope.com/u/Smeghead)\ **Post date:** [April 21, 2014, 8:59pm UTC](https://boards.straightdope.com/t/is-there-a-name-for-this-effect-in-scientific-publishing/686630/6 "2014-04-21T20:59:35Z")
The easiest way around this is a [Bonferroni correction.](http://en.wikipedia.org/wiki/Bonferroni_correction) Every paper I’ve seen that does large numbers of tests uses this or something similar to account for the problem. ---
**Author:** ![KarlGauss](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/karlgauss/32/3713_2.png) [@KarlGauss](https://boards.straightdope.com/u/KarlGauss)\ **Post date:** [April 21, 2014, 9:07pm UTC](https://boards.straightdope.com/t/is-there-a-name-for-this-effect-in-scientific-publishing/686630/7 "2014-04-21T21:07:40Z")
The problem is sometimes referred to as the [multiple comparisons problem](https://en.wikipedia.org/wiki/Multiple_comparisons_problem), or even the [multiple comparisons fallacy](http://www.fallacyfiles.org/multcomp.html). ---
**Author:** ![Colibri](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/colibri/32/1841_2.png) [@Colibri](https://boards.straightdope.com/u/Colibri)\ **Post date:** [April 21, 2014, 9:14pm UTC](https://boards.straightdope.com/t/is-there-a-name-for-this-effect-in-scientific-publishing/686630/8 "2014-04-21T21:14:16Z")
> [@Budget\_Player\_Cadet](#): > > Essentially publishing so many randomized controlled trials that by sheer probability, something incredibly unlikely is almost certain to come out (note how even with P = 0,95 of a correct result, by the end of the 21 tests, the chances of no false positives is still below 50%). It should be mentioned that scientists do take the level of significance into account when evaluating a paper. If you report a highly controversial result, you’ll have a lot more credibility if your results are significant at the .001 level than if you barely crack .05. Another way of dealing with the same problem is to conduct a [meta-analysis](http://en.wikipedia.org/wiki/Meta-analysis), combining the results of many different studies to evaluate the effect of sample size. ---
**Author:** ![D18](https://avatars.discourse-cdn.com/v4/letter/d/c57346/32.png) [@D18](https://boards.straightdope.com/u/D18)\ **Post date:** [April 21, 2014, 9:14pm UTC](https://boards.straightdope.com/t/is-there-a-name-for-this-effect-in-scientific-publishing/686630/9 "2014-04-21T21:14:56Z")
I believe the specific scenario in the comic is called [publication bias](http://en.m.wikipedia.org/wiki/Publication_bias). ---
**Author:** ![Cerealbox](https://avatars.discourse-cdn.com/v4/letter/c/7ab992/32.png) [@Cerealbox](https://boards.straightdope.com/u/Cerealbox)\ **Post date:** [April 21, 2014, 9:24pm UTC](https://boards.straightdope.com/t/is-there-a-name-for-this-effect-in-scientific-publishing/686630/10 "2014-04-21T21:24:29Z")
> [@D18](#): > > I believe the specific scenario in the comic is called [publication bias](http://en.m.wikipedia.org/wiki/Publication_bias). Bingo. ---
**Author:** ![Cerealbox](https://avatars.discourse-cdn.com/v4/letter/c/7ab992/32.png) [@Cerealbox](https://boards.straightdope.com/u/Cerealbox)\ **Post date:** [April 21, 2014, 9:28pm UTC](https://boards.straightdope.com/t/is-there-a-name-for-this-effect-in-scientific-publishing/686630/11 "2014-04-21T21:28:14Z")
Actually, if the comic is referring to non-scientific publushing, it could be _double_ publication bias. ---
**Author:** ![Colibri](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/colibri/32/1841_2.png) [@Colibri](https://boards.straightdope.com/u/Colibri)\ **Post date:** [April 21, 2014, 9:30pm UTC](https://boards.straightdope.com/t/is-there-a-name-for-this-effect-in-scientific-publishing/686630/12 "2014-04-21T21:30:31Z")
> [@D18](#): > > I believe the specific scenario in the comic is called [publication bias](http://en.m.wikipedia.org/wiki/Publication_bias). However, it’s quite possible that the media would seize on the one significant outcome even if the paper itself had included all of them. ---
**Author:** ![dstarfire](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/dstarfire/32/5762_2.png) [@dstarfire](https://boards.straightdope.com/u/dstarfire)\ **Post date:** [April 21, 2014, 10:11pm UTC](https://boards.straightdope.com/t/is-there-a-name-for-this-effect-in-scientific-publishing/686630/13 "2014-04-21T22:11:56Z")
I’ve heard it called cherry picking, if (at some point) the only study mentioned is the odd one. ---
**Author:** ![wolfpup](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/wolfpup/32/10618_2.png) [@wolfpup](https://boards.straightdope.com/u/wolfpup)\ **Post date:** [April 21, 2014, 10:13pm UTC](https://boards.straightdope.com/t/is-there-a-name-for-this-effect-in-scientific-publishing/686630/14 "2014-04-21T22:13:07Z")
> [@Budget\_Player\_Cadet](#): > > Is there a name for this? Yes, the problem is called “pharmaceutical research” (which is closely related to studies in substance safety such as whether jelly beans cause acne). 😃 I am being somewhat facetious, but not entirely. For example it’s been known for some time that the [Kaplan-Meier](http://en.wikipedia.org/wiki/Kaplan%E2%80%93Meier_estimator) efficacy of certain drugs can suddenly and mysteriously reverse itself, and no one knows why. These strangenesses are so pronounced in medicine that the Harvard Medical School has set up [an entire program](http://programinplacebostudies.org/) to study them. Only peripherally related to your question, I know, but still… seems related to the phenomenon of interactions being so profoundly complex that it can be difficult to study them systematically or describe them in statistically reliable terms. ---
**Author:** ![Budget\_Player\_Cadet](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/budget_player_cadet/32/205_2.png) [@Budget\_Player\_Cadet](https://boards.straightdope.com/u/Budget_Player_Cadet)\ **Post date:** [April 21, 2014, 11:45pm UTC](https://boards.straightdope.com/t/is-there-a-name-for-this-effect-in-scientific-publishing/686630/15 "2014-04-21T23:45:52Z")
> [@Colibri](#): > > It’s called a [Type I error](http://en.wikipedia.org/wiki/Type_I_and_type_II_errors) (false positive). Reviewers do consider this when there are a large number of tests being conducted. > > Note that a Type I error can occur even with a single test, it’s just that the more tests there are the more probable they will occur. Yeah, it’s a type I error, but I mean specifically testing so often to try to get a handful of positive flukes that you can wave around. Possibly maliciously, possibly unintentionally but without paying attention to it. > [@Colibri](#): > > It should be mentioned that scientists do take the level of significance into account when evaluating a paper. If you report a highly controversial result, you’ll have a lot more credibility if your results are significant at the .001 level than if you barely crack .05. > > Another way of dealing with the same problem is to conduct a [meta-analysis](http://en.wikipedia.org/wiki/Meta-analysis), combining the results of many different studies to evaluate the effect of sample size. Oh, I know how to mitigate it, I just want to make sure there isn’t already some term for it before I make up a term for it on my blog and look like a tool. 😃 ---
**Author:** ![Isilder](https://avatars.discourse-cdn.com/v4/letter/i/cc9497/32.png) [@Isilder](https://boards.straightdope.com/u/Isilder)\ **Post date:** [April 22, 2014, 1:04am UTC](https://boards.straightdope.com/t/is-there-a-name-for-this-effect-in-scientific-publishing/686630/16 "2014-04-22T01:04:33Z")
> [@D18](#): > > I believe the specific scenario in the comic is called [publication bias](http://en.m.wikipedia.org/wiki/Publication_bias). Not quite. There is only one paper about food dye XYZ (assume thats the green dye… ) to publish ! Publication bias is where there are a number of other papers said food dye XYZ does NOT cause cancer, but only the paper that gives a “positive” is published and hence the journals effectively say that “1 out of 1 papers says green colour XYZ does cause cancer”. If you included non-published papers, it may be that actually 4 out of 5 papers say that it doesn’t. The slide is not actually showing any error - it may well be true. How else can a ‘green dye XYZ causes cancer’ study be initiated unless someone found some reason to target that ? Of course, the next study may be a meta-study of other research done on food dye XYZ … hopefully that meta-study is not biased due to publication bias… ---
**Author:** ![Colibri](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/colibri/32/1841_2.png) [@Colibri](https://boards.straightdope.com/u/Colibri)\ **Post date:** [April 22, 2014, 2:36am UTC](https://boards.straightdope.com/t/is-there-a-name-for-this-effect-in-scientific-publishing/686630/17 "2014-04-22T02:36:52Z")
> [@Budget\_Player\_Cadet](#): > > Yeah, it’s a type I error, but I mean specifically testing so often to try to get a handful of positive flukes that you can wave around. Possibly maliciously, possibly unintentionally but without paying attention to it. If you’re doing it deliberately, the best term for it would be “scientific fraud.” The problem is discussed in basic statistics courses, so if you’re doing it unintentionally the best term would be “scientific incompetence.” This is for the case of a single investigator or team of investigators doing all of the tests (as in the case illustrated in xckd). If different investigators are doing single tests, and only the ones with positive results publish, then it’s a case of publication bias. ---
**Author:** ![Hector\_St\_Clare](https://avatars.discourse-cdn.com/v4/letter/h/b5ac83/32.png) [@Hector\_St\_Clare](https://boards.straightdope.com/u/Hector_St_Clare)\ **Post date:** [April 24, 2014, 2:25am UTC](https://boards.straightdope.com/t/is-there-a-name-for-this-effect-in-scientific-publishing/686630/18 "2014-04-24T02:25:57Z")
What you’re talking about is the problem of multiple comparisons. > **[Multiple comparisons problem](https://en.wikipedia.org/wiki/Multiple_comparisons_problem)** > > In statistics, the multiple comparisons, multiplicity or multiple testing problem occurs when one considers a set of statistical inferences simultaneously or infers a subset of parameters selected based on the observed values. > The more inferences are made, the more likely erroneous inferences become. Several statistical techniques have been developed to address that problem, typically by requiring a stricter significance threshold for individual comparisons, so as to compensate for the number o... I like to describe it more picturesquely (and if I’m ever called upon to do an intro stats lecture, I will) as “if you throw enough turds at a wall, one of them will stick.” The problem has been known about since forever, and is dealt with in introductory stats classes, so a researcher really should have no excuse for not doing multiple comparisons corrections. At one level, there are a lot of ways to ‘solve’ the problem: you can simply increase the threshold of statistical significance, or you can constrain the number of hypotheses you test, or some combination of the two. I like to use the Bonferroni correction, which **smeghead** alludes to: it’s the oldest, simplest, easiest to use, one of the most general and IIRC the most ‘conservative’ procedure. There are lots of other ways to correct for multiple comparisons, depending on the exact design of your experiment (Sidak, Tukey, Dunnett, Scheffe, false discovery rate, Hochberg, and others), and some of them can be more powerful. At another level, the problem is a thorny one because typically, you ‘protect’ against type I error at the cost of increasing type II error (draw some normal distributions and you will immediately see graphically why this is the case). Andrew Gelman recently had a blog post where he points out that typical multiple comparisons corrections have another problem: effect size and level of significance are typically not independent (in a t-test, for example, the level of significance is going to depend on effect size, standard error and degrees of freedom), so by increasing the threshold of statistical significance, you’re going to ensure that only very large effect sizes make it through your testing procedure, which is going to distort your estimates about how large the true effects are. Regardless, you need to do _some_ correction for multiple comparisons (and really, you should address the problem in advance by running experiments with larger sample sizes, well specified models, and clearly designed hypotheses rather than running fishing expeditions to see what looks interesting). Just my two cents, so take it for what its’ worth, i.e. not much. ---
**Author:** ![KarlGauss](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/karlgauss/32/3713_2.png) [@KarlGauss](https://boards.straightdope.com/u/KarlGauss)\ **Post date:** [April 24, 2014, 3:06am UTC](https://boards.straightdope.com/t/is-there-a-name-for-this-effect-in-scientific-publishing/686630/19 "2014-04-24T03:06:18Z")
> [@Hector\_St\_Clare](#): > > . . . clearly designed hypotheses rather than running fishing expeditions to see what looks interesting). . . This merits emphasis. In other words, if comparisons are being made here, there, and everywhere, finding a significant difference, as has been pointed out, is expected and likely of no import. On the other hand if the comparisons to be done were specified in advance (and, yes, limited in number) and were formed the basis of some _a priori_ hypotheses or expectations, the result is much more telling. Such advance specification is especially important when subgroups are being compared. If the subgroups were defined before the study, and if there was a plausible or justifiable reason to form such a subgroup in the first place, comparisons that are then found to reach statistical significance can be very illuminating, unlike those formed _post hoc_. ---
**Author:** ![mcgato](https://avatars.discourse-cdn.com/v4/letter/m/ac8455/32.png) [@mcgato](https://boards.straightdope.com/u/mcgato)\ **Post date:** [April 24, 2014, 1:11pm UTC](https://boards.straightdope.com/t/is-there-a-name-for-this-effect-in-scientific-publishing/686630/20 "2014-04-24T13:11:11Z")
I think that it shows the difference between testwise error rate and experimentwise error rate. In the case of the comic, the testwise error rate is 0.05 (which relates to the 95% confidence level). So if the effect is not significant, the test will erroneously conclude that it is significant 5% of the time. But doing 20 tests, each with a testwise error rate of 0.05, will lead to an experimentwise error rate that is much higher. [The equation is found here.](http://davidmlane.com/hyperstat/A43646.html) For the comic, the 20 tests would result in an experimentwise error rate of about 64%. So it is more likely than not that they will erroneously conclude that something is significant.