# Is Snopes wrong? (Statistical Significance) **URL:** \ **Category:** Factual Questions\ **Created:** [December 27, 2015, 8:38pm UTC](https://boards.straightdope.com/t/is-snopes-wrong-statistical-significance/741487 "2015-12-27T20:38:06Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![thelurkinghorror](https://avatars.discourse-cdn.com/v4/letter/t/7c8e57/32.png) [@thelurkinghorror](https://boards.straightdope.com/u/thelurkinghorror)\ **Post date:** [December 28, 2015, 7:04am UTC](https://boards.straightdope.com/t/is-snopes-wrong-statistical-significance/741487/21 "2015-12-28T07:04:53Z")
> [@dasmoocher](#): > > Related to sensational headlines, here’s a closer blog look at a recent (12/21/2015) one: > [Skepticism About Study Linking Antidepressants And Autism](http://www.forbes.com/sites/stevensalzberg/2015/12/21/skepticism-about-study-linking-antidepressants-and-autism/) A recent well known story came out “BACON CAUSES CANCER!!!” …well processed meat, but bacon is the trendy one… …well only one type of cancer… …[well it was already a low chance to start with, and even with increased risk is still low](http://scienceblog.cancerresearchuk.org/2015/10/26/processed-meat-and-cancer-what-you-need-to-know/). ---
**Author:** ![coremelt](https://avatars.discourse-cdn.com/v4/letter/c/4491bb/32.png) [@coremelt](https://boards.straightdope.com/u/coremelt)\ **Post date:** [December 28, 2015, 7:48am UTC](https://boards.straightdope.com/t/is-snopes-wrong-statistical-significance/741487/22 "2015-12-28T07:48:13Z")
Well lets address the original issue. There is very little controversy (In Australia) that all gun deaths have gone down considerably in Australia since the 1997 buyback so whether that one statistic of the gun crime rate in Victoria from 1996 to 1997 is valid or not is a moot point. Study: [http://sydney.edu.au/news/84.html?newsstoryid=1502](http://sydney.edu.au/news/84.html?newsstoryid=1502) ---
**Author:** ![scr4](https://avatars.discourse-cdn.com/v4/letter/s/59ef9b/32.png) [@scr4](https://boards.straightdope.com/u/scr4)\ **Post date:** [December 28, 2015, 11:36am UTC](https://boards.straightdope.com/t/is-snopes-wrong-statistical-significance/741487/23 "2015-12-28T11:36:31Z")
> [@OldGuy](#): > > If you assume a Poisson distribution with a mean of 7 deaths, then the probability of 17 or more deaths in a year is 0.096% which would generally be considered significant. > > If you assume a mean of 19 deaths then the probability of 7 or fewer is 0.543% also generally considered significant. > > If you assume a mean of 13 (the average of 7 and 19) then you’d have 7 or fewer 5.4% of the time and 17 or more 16.45% of the time. Neither of those is particularly significant at standard levels. > > So it all depends on how you look at it. I think the conclusion to draw from this is, if we only have those 2 data points to work with, then we’d conclude that there is no evidence for a statistically significant increase between these 2 years, because the 3rd possibility above is, well, a possibility. ---
**Author:** ![lazybratsche](https://avatars.discourse-cdn.com/v4/letter/l/ba8739/32.png) [@lazybratsche](https://boards.straightdope.com/u/lazybratsche)\ **Post date:** [December 28, 2015, 3:25pm UTC](https://boards.straightdope.com/t/is-snopes-wrong-statistical-significance/741487/24 "2015-12-28T15:25:44Z")
We can model this data as coin flips, and apply the appropriate statistical test. E.g. every year each person flips an extremely biased coin to determine whether they are victims of homicide by firearm. In this case the exact [binomial test](https://en.wikipedia.org/wiki/Binomial_test) is the best option\*. Specifically: Given a rate of 7 events per 4,500,000 trials, what is the probability of observing 19 or more events in 4,500,000 trials? Plugging the numbers in R, I get p = 0.00013. That would be statistically significant by any reasonable definition. However, since these particular numbers were cherry picked to support a specific political agenda, we really can’t take them at face value. If you dig through any perfectly random data you will always be able to find some extreme comparisons, especially if you start combing through subset. “In East BFE, Victoria, there were 0 firearms homicides in 1996 and 1 in 1997! The homicide rate went up INFINITY percent!!!1!” To really get to the heart of the matter, what we really want to know is whether gun control reduces gun violence. Since we do not have a control Australia that is identical to real-Australia in every way except this legislation, we can never have a completely unambiguous answer. However, a more thorough analysis of firearm-related crime rates, using all the data and rigorous statistical methods, would be infinitely more informative than bullshit cherry picked examples. But that’s way more work than I’m willing to do in my free time! \*The t-test or ANOVA is only appropriate for comparing the means of normally distributed data. That would be useful here only if we had data from multiple years before and after some event of interest. Even then, the homicides per year may not be normally distributed, and we’d be better off using a test that assumes a more appropriate distribution or no distribution at all. ---
**Author:** ![El\_Zagna](https://avatars.discourse-cdn.com/v4/letter/e/d2c977/32.png) [@El\_Zagna](https://boards.straightdope.com/u/El_Zagna)\ **Post date:** [December 28, 2015, 4:20pm UTC](https://boards.straightdope.com/t/is-snopes-wrong-statistical-significance/741487/25 "2015-12-28T16:20:19Z")
> [@lazybratsche](#): > > We can model this data as coin flips, and apply the appropriate statistical test. E.g. every year each person flips an extremely biased coin to determine whether they are victims of homicide by firearm. In this case the exact [binomial test](https://en.wikipedia.org/wiki/Binomial_test) is the best option\*. > > Specifically: > > Given a rate of 7 events per 4,500,000 trials, what is the probability of observing 19 or more events in 4,500,000 trials? Plugging the numbers in R, I get p = 0.00013. That would be statistically significant by any reasonable definition. Ah, excellent, **lazybratsche**. And we’re back to the OP. I appreciate everyone’s caveats about the \*practical \*significance of using these numbers, but remember this is GQ not GD, and the specific question is whether or not the numbers are \*statistically \*significant. I took this question over to a [statistics forum](http://www.talkstats.com/showthread.php/63467-Is-Snopes-wrong-%28Statistical-Significance%29), and so far the only answer I got back seemed to confirm what \*\*lazybratsche \*\*is saying: > [@](#): > > …if the null hypothesis was that the n=26 homicides were distributed evenly between years, then a one-way Chi² test would result in p =0.031 [continuity corrected], i.e. a statistically significant increase. Some of the terms are starting to come back to me. He used a Chi² test, but that’s a version of an exact binomial test isn’t it? ---
**Author:** ![Lemur866](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/lemur866/32/434_2.png) [@Lemur866](https://boards.straightdope.com/u/Lemur866)\ **Post date:** [December 28, 2015, 5:18pm UTC](https://boards.straightdope.com/t/is-snopes-wrong-statistical-significance/741487/26 "2015-12-28T17:18:20Z")
Also remember that a 95% confidence level means that one time in 20 you’re going to get the wrong result. That’s fine for any one test, but it means that when you do lots of significance tests lots of them are going to be wrong by chance. Out of 100 papers presenting statistically significant results, 5 of those papers are going to be incorrect. ---
**Author:** ![thelurkinghorror](https://avatars.discourse-cdn.com/v4/letter/t/7c8e57/32.png) [@thelurkinghorror](https://boards.straightdope.com/u/thelurkinghorror)\ **Post date:** [December 28, 2015, 5:59pm UTC](https://boards.straightdope.com/t/is-snopes-wrong-statistical-significance/741487/27 "2015-12-28T17:59:59Z")
> [@Lemur866](#): > > Also remember that a 95% confidence level means that one time in 20 you’re going to get the wrong result. That’s fine for any one test, but it means that when you do lots of significance tests lots of them are going to be wrong by chance. Out of 100 papers presenting statistically significant results, 5 of those papers are going to be incorrect. That’s true if an effect/difference on fact exists in reality. If it doesn’t, your chance of type I error is zero, but your chance of type ii can be greater than 5% (inverse of power, which can be improved through several methods). ---
**Author:** ![Pasta](https://avatars.discourse-cdn.com/v4/letter/p/ecccb3/32.png) [@Pasta](https://boards.straightdope.com/u/Pasta)\ **Post date:** [December 28, 2015, 6:07pm UTC](https://boards.straightdope.com/t/is-snopes-wrong-statistical-significance/741487/28 "2015-12-28T18:07:43Z")
Something that hasn’t been said explicitly: In the OP’s example, the exact population size is irrelevant, to good approximation. That is, the significance of an additional 12 homicides is the same to many decimal places whether the underlying population is 4.5 million, 9 million, or 1 billion. As long as the population is much larger than the number of homicides itself, then it’s just the number of homicides you need to look at. (This is exactly the Poisson limit). ---
**Author:** ![Pasta](https://avatars.discourse-cdn.com/v4/letter/p/ecccb3/32.png) [@Pasta](https://boards.straightdope.com/u/Pasta)\ **Post date:** [December 28, 2015, 6:36pm UTC](https://boards.straightdope.com/t/is-snopes-wrong-statistical-significance/741487/29 "2015-12-28T18:36:02Z")
> [@OldGuy](#): > > If you assume a Poisson distribution with a mean of 7 deaths, then the probability of 17 or more deaths in a year is 0.096% which would generally be considered significant. > > If you assume a mean of 19 deaths then the probability of 7 or fewer is 0.543% also generally considered significant. > > If you assume a mean of 13 (the average of 7 and 19) then you’d have 7 or fewer 5.4% of the time and 17 or more 16.45% of the time. Neither of those is particularly significant at standard levels. > > So it all depends on how you look at it. The last way is closest to how one should test this, though it’s not quite complete. One needs to order the possible sets of outcomes and integrate (sum) the probabilities over all worse cases. In frequentist statistics the ordering principle is a matter of choice, but using an obvious ordering for this discrete problem: We observe 26 homicides. If each year truly has the same expectation for the number of homicides, we need to find the sum of the probabilities for all possible “splits” that are as unlikely as, or _more_ unlikely than, the split (7, 19) we observe. There are only 27 possible splits. The ones as or more unlikely than (7, 19) have a cumulative probability of 2.9%. Or flipping the language around: **There is a 97.1% chance that the two years would match better than they did if the years were truly equivalent.** It becomes an exercise for the reader to decide if this probability is significant to them. ---
**Author:** ![Riemann](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/riemann/32/3133_2.png) [@Riemann](https://boards.straightdope.com/u/Riemann)\ **Post date:** [December 28, 2015, 7:16pm UTC](https://boards.straightdope.com/t/is-snopes-wrong-statistical-significance/741487/30 "2015-12-28T19:16:41Z")
**Pasta** ’s test looks right to me. However, I’m concerned about one of the OP’s earlier comments. > [@El\_Zagna](#): > > Hmmm… Just to be clear, this is a question about pure, plug-numbers-into-a-formula, statistical significance. Forget that it’s about homicides… There is **no such thing** as “pure, plug-numbers-into-a-formula, statistical significance”. It is an **intrinsic** part of any statistical analysis that you look at the entire picture to make sure that your analysis is valid. In this particular case, the two things that leap out at me are: (1) Were the data cherry-picked? **Pasta** ’s test is saying: \*Under the null hypothesis of a random Poisson process, you expect to see these data, or something more extreme, less than 3% of the time. \* So, where did these data come from? If, for example, there are data for 10 states over 5 years, you would expect to find at least one such occurrence. So if these data came from a motivated person who went looking for them, the result is meaningless. You simply **cannot test for significance** unless you know how these particular data points were selected for analysis. Unless you know that, there is literally **no such thing** as a statistical test for significance. (2) Are the model assumptions reasonable? A Poisson process is the obvious model here. But the assumption of that model is that events are **independent**. This is almost certainly quite badly wrong for homicides: in some cases, there will have been multiple deaths in a single incident. This kind of deviation from Poisson is known as “clumping”. It’s obvious intuitively that clumping will make a wider spread of results more likely. That’s why, when I suggested the model, I suggested it as giving a **bound** on significance. If, under Poisson, the result is **not** significant, then I think that’s probably a valid result, because any deviation from Poisson is almost certainly clumping. But if you get significance under Poisson you have more work to do to analyze the degree of clumping, and possibly find a better model to decide if the result is valid. ---
**Author:** ![dasmoocher](https://avatars.discourse-cdn.com/v4/letter/d/dbc845/32.png) [@dasmoocher](https://boards.straightdope.com/u/dasmoocher)\ **Post date:** [December 28, 2015, 7:53pm UTC](https://boards.straightdope.com/t/is-snopes-wrong-statistical-significance/741487/31 "2015-12-28T19:53:42Z")
> [@Lemur866](#): > > Also remember that a 95% confidence level means that one time in 20 you’re going to get the wrong result. That’s fine for any one test, but it means that when you do lots of significance tests lots of them are going to be wrong by chance. Green jelly beans cause acne: [xkcd: Significant](https://xkcd.com/882/) ---
**Author:** ![Riemann](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/riemann/32/3133_2.png) [@Riemann](https://boards.straightdope.com/u/Riemann)\ **Post date:** [December 28, 2015, 8:22pm UTC](https://boards.straightdope.com/t/is-snopes-wrong-statistical-significance/741487/32 "2015-12-28T20:22:21Z")
> [@dasmoocher](#): > > Green jelly beans cause acne: > [xkcd: Significant](https://xkcd.com/882/) And this is not just an xkcd joke. > [@](#): > > In the biggest project of its kind… 269 co-authors repeated work reported in 98 original papers from three psychology journals, to see if they independently came up with the same results…Whereas 97% of the original studies found a significant effect, only 36% of replication studies found significant results. [http://www.nature.com/news/over-half-of-psychology-studies-fail-reproducibility-test-1.18248](http://www.nature.com/news/over-half-of-psychology-studies-fail-reproducibility-test-1.18248) The problem is that the multiple hypothesis testing illustrated in that xkcd cartoon is so often hiddden. Even if you’re honest, it’s so easy to fool _yourself_. Frequentist hypothesis testing for significance as a _post hoc_ measure of whether we should pay attention to something is just so badly flawed that it’s completely unreliable. Unless somebody clearly states and publishes all their hypotheses beforehand, before they do any experiments, I think it’s almost meaningless. ---
**Author:** ![El\_Zagna](https://avatars.discourse-cdn.com/v4/letter/e/d2c977/32.png) [@El\_Zagna](https://boards.straightdope.com/u/El_Zagna)\ **Post date:** [December 28, 2015, 10:21pm UTC](https://boards.straightdope.com/t/is-snopes-wrong-statistical-significance/741487/33 "2015-12-28T22:21:01Z")
> [@Riemann](#): > > There is **no such thing** as “pure, plug-numbers-into-a-formula, statistical significance”. It is an **intrinsic** part of any statistical analysis that you look at the entire picture to make sure that your analysis is valid. Not in this case. I carefully limited the question to the claim of _statistical significance_, and even restated it to eliminate the context. Questions about the methodology or the validity of the numbers is a different part of the conversation. You are trying to take this into a realm that I specifically wanted to avoid. ---
**Author:** ![Isilder](https://avatars.discourse-cdn.com/v4/letter/i/cc9497/32.png) [@Isilder](https://boards.straightdope.com/u/Isilder)\ **Post date:** [December 28, 2015, 10:37pm UTC](https://boards.straightdope.com/t/is-snopes-wrong-statistical-significance/741487/34 "2015-12-28T22:37:30Z")
> [@El\_Zagna](#): > > Snopes addresses the Australian gun buyback program here: [Australian Guns](http://www.snopes.com/crime/statistics/ausguns.asp). In it they make a statement that doesn’t seem right to me: > > My statistics is pretty rusty, but my gut is telling me that this is incorrect. The sample size is 4.5 million, so a 171% increase of something should be statistically significant. I think. > > Remind me how this works. Typically by looking at the credentials of the person giving you the analysis (without the raw data… . Whats wrong in this case ? Bizarre subdivision . Taking too small a subset, for the purpose of saying there is no data. This just like saying that a bar of chocolate has 500 servicing, and there’s 0 grams of fat per serving… Snopes takes a small dataset, then says it is too small to use. Raw data [http://www.gunpolicy.org/firearms/region/australia](http://www.gunpolicy.org/firearms/region/australia) Sample size obviously large enough, the change during the 1990’s is obvious ? Well sort of … Random breath testing, and “responsible serving of alcohol” policy, may have something to do with it, because its clear that the number of pubs has reduced … and anyway the drunk person is far less likely to have their own car nearby, and so no gun nearby. The Australian gun control was to reduce massacre events, and that has worked. It has also protected native wildlife, and increases the longevity of road signs. ( Maybe the ball bearing bird moved off somewhere else ? [https://www.youtube.com/watch?v=Pt8-HXJ5oh8](https://www.youtube.com/watch?v=Pt8-HXJ5oh8) … obviously a drinkers song 🙂 ) ---
**Author:** ![Riemann](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/riemann/32/3133_2.png) [@Riemann](https://boards.straightdope.com/u/Riemann)\ **Post date:** [December 28, 2015, 10:38pm UTC](https://boards.straightdope.com/t/is-snopes-wrong-statistical-significance/741487/35 "2015-12-28T22:38:34Z")
> [@El\_Zagna](#): > > Not in this case. I carefully limited the question to the claim of _statistical significance_, and even restated it to eliminate the context. Questions about the methodology or the validity of the numbers is a different part of the conversation. You are trying to take this into a realm that I specifically wanted to avoid. Again, there is **no such thing** as a test for statistical significance without context. A significance test involves (at least) **all** of the following steps (1) How were the data obtained? (2) What is the appropriate model, does the phenomenon satisfy the assumptions of the model? (3) How do I do the math to perform a test under this model? In your OP, you had not addressed (1) or (2). If your OP had said, “I know that these data were randomly selected in an unbiased fashon; and I know that the phenomenon satisfies the assumptions of the Poisson model. How do I do perform a significance test under Poisson?” Then, and only then, it would reasonable to respond with just (3). It is not nitpicking, it is simply **wrong** to claim that anything represents a significance test without addressing (1) and (2). Even if you ignore (1), you need to know the appropriate model, because there are dozens of different statisticals tests applicable to different models and different _kinds_ of data. ---
**Author:** ![Pasta](https://avatars.discourse-cdn.com/v4/letter/p/ecccb3/32.png) [@Pasta](https://boards.straightdope.com/u/Pasta)\ **Post date:** [December 28, 2015, 10:46pm UTC](https://boards.straightdope.com/t/is-snopes-wrong-statistical-significance/741487/36 "2015-12-28T22:46:05Z")
> [@El\_Zagna](#): > > Not in this case. I carefully limited the question to the claim of _statistical significance_, and even restated it to eliminate the context. Questions about the methodology or the validity of the numbers is a different part of the conversation. You are trying to take this into a realm that I specifically wanted to avoid. **Riemann** is on point. Taking your previous clarification: > [@El Zagna](#): > > Just to be clear, this is a question about pure, plug-numbers-into-a-formula, statistical significance. Forget that it’s about homicides. You can’t forget that it’s about homicides. What if the thing being counted was “out of all coins returned as change over the course of a year in the U.S., how many were returned to people named Joey McJoe on Saturdays in Wisconsin?” If in two sequential years we saw 7 coins returned and 19 coins returned, is that difference significant? Even just calculating the probability of that occurring requires us to realize that change gets returned in shots with multiple coins coming at once, and that certain numbers of coins per instance are more likely than others. If instead of all coins we asked about pennies specifically, we’d get yet a different answer. Even with a specific probability in hand under a specific model or models, it’s a separate step entirely to assess the significance of it. Here, too, you cannot divorce yourself from the actual system at hand. If the homicide problem was “After looking at 30 countries where good homicide statistics were available, we found that Australia had the most extreme deviation (p=2.9%),” then 2.9% would not be significant by any reasonable measure since there were 30 trials. None of this is to say that you haven’t provided some context – you have. But it is incorrect to say that that context doesn’t enter when establishing the statistical significance of 7 versus 19 of anything. The context is crucial. ---
**Author:** ![Isilder](https://avatars.discourse-cdn.com/v4/letter/i/cc9497/32.png) [@Isilder](https://boards.straightdope.com/u/Isilder)\ **Post date:** [December 28, 2015, 10:56pm UTC](https://boards.straightdope.com/t/is-snopes-wrong-statistical-significance/741487/37 "2015-12-28T22:56:47Z")
> [@Isilder](#): > > The Australian gun control was to reduce massacre events, and that has worked. Sorry can’t say that, because it is too small a past incidence rate compared to population size… The appropriate way to measure it is massacres using civilian owned rifles per century or something… All this overanalysis… Obviously when the gun related murder rate is 200 to 300 a year, we have enough data… You all fell for snopes bizarre … bizarre what ? its just bizarre talk. ---
**Author:** ![Riemann](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/riemann/32/3133_2.png) [@Riemann](https://boards.straightdope.com/u/Riemann)\ **Post date:** [December 28, 2015, 11:00pm UTC](https://boards.straightdope.com/t/is-snopes-wrong-statistical-significance/741487/38 "2015-12-28T23:00:26Z")
Just to add: the burden of proof is obviously to _show_ significance. This includes _showing_ that the data sampling meets the assumptions of the statistical test. If there is no information available about how the data were chosen, then Snopes is precisely correct to say the data are not statistically significant, and this does not require math. ---
**Author:** ![Isilder](https://avatars.discourse-cdn.com/v4/letter/i/cc9497/32.png) [@Isilder](https://boards.straightdope.com/u/Isilder)\ **Post date:** [December 28, 2015, 11:11pm UTC](https://boards.straightdope.com/t/is-snopes-wrong-statistical-significance/741487/39 "2015-12-28T23:11:57Z")
> [@Isilder](#): > > You all fell for snopes bizarre … bizarre what ? its just bizarre talk. Ok I read snopes carefully Introduction… introduces some ways to make a fallacious argument from statistics. hmm that is propaganda, because it begs the question of whether they were relevant. This isn’t a page from a text book, it is meant to be a small piece which would probably be strictly on topic, OR make it clear that any irrelevant section is irrelevant. Body… quotes some statistics, but its a real jig saw of random facts. How do they fit together ? To what falacial argument do they address ? More propaganda, by demonstrating how a single year in Victoria is too small a sample size, it was left to the reader to decide if that could be extended to the entire country over the TWO DECADES ? You all fell for basic propaganda and went on to describe Poisson… But true , there can be other causes, because the “getting drunk at the pub every afternoon” has gone away with random breath testing and taxes, and that is good for removing the guns from drunk people, and for the populations brains .However its not clear that is so good, because it may mean the boys with pistols are less socialised - they are fearful of strangers and have less experience with conflict resolution - maybe they turn to pistols to communicate their fears… instead of talking to the other side to diffuse the situation, negotiate a peace treaty… ---
**Author:** ![Riemann](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/riemann/32/3133_2.png) [@Riemann](https://boards.straightdope.com/u/Riemann)\ **Post date:** [December 28, 2015, 11:22pm UTC](https://boards.straightdope.com/t/is-snopes-wrong-statistical-significance/741487/40 "2015-12-28T23:22:02Z")
> [@Isilder](#): > > You all fell for basic propaganda and went on to describe Poisson… I didn’t read the Snopes page. I don’t even know what the political background is. I think most of the responders here (at least at the start) were just discussing how to model homicide rates, and what is required for a valid significance test. [Previous page](https://boards.straightdope.com/t/is-snopes-wrong-statistical-significance/741487.md?page=1) [Next page](https://boards.straightdope.com/t/is-snopes-wrong-statistical-significance/741487.md?page=3)