A recent well known story came out “BACON CAUSES CANCER!!!”
…well processed meat, but bacon is the trendy one…
…well only one type of cancer…
…well it was already a low chance to start with, and even with increased risk is still low.
Well lets address the original issue. There is very little controversy (In Australia) that all gun deaths have gone down considerably in Australia since the 1997 buyback so whether that one statistic of the gun crime rate in Victoria from 1996 to 1997 is valid or not is a moot point.
I think the conclusion to draw from this is, if we only have those 2 data points to work with, then we’d conclude that there is no evidence for a statistically significant increase between these 2 years, because the 3rd possibility above is, well, a possibility.
We can model this data as coin flips, and apply the appropriate statistical test. E.g. every year each person flips an extremely biased coin to determine whether they are victims of homicide by firearm. In this case the exact binomial test is the best option*.
Specifically:
Given a rate of 7 events per 4,500,000 trials, what is the probability of observing 19 or more events in 4,500,000 trials? Plugging the numbers in R, I get p = 0.00013. That would be statistically significant by any reasonable definition.
However, since these particular numbers were cherry picked to support a specific political agenda, we really can’t take them at face value. If you dig through any perfectly random data you will always be able to find some extreme comparisons, especially if you start combing through subset. “In East BFE, Victoria, there were 0 firearms homicides in 1996 and 1 in 1997! The homicide rate went up INFINITY percent!!!1!”
To really get to the heart of the matter, what we really want to know is whether gun control reduces gun violence. Since we do not have a control Australia that is identical to real-Australia in every way except this legislation, we can never have a completely unambiguous answer. However, a more thorough analysis of firearm-related crime rates, using all the data and rigorous statistical methods, would be infinitely more informative than bullshit cherry picked examples.
But that’s way more work than I’m willing to do in my free time!
*The t-test or ANOVA is only appropriate for comparing the means of normally distributed data. That would be useful here only if we had data from multiple years before and after some event of interest. Even then, the homicides per year may not be normally distributed, and we’d be better off using a test that assumes a more appropriate distribution or no distribution at all.
Ah, excellent, lazybratsche. And we’re back to the OP. I appreciate everyone’s caveats about the *practical *significance of using these numbers, but remember this is GQ not GD, and the specific question is whether or not the numbers are *statistically *significant.
I took this question over to a statistics forum, and so far the only answer I got back seemed to confirm what **lazybratsche **is saying:
Some of the terms are starting to come back to me. He used a Chi² test, but that’s a version of an exact binomial test isn’t it?
Also remember that a 95% confidence level means that one time in 20 you’re going to get the wrong result. That’s fine for any one test, but it means that when you do lots of significance tests lots of them are going to be wrong by chance. Out of 100 papers presenting statistically significant results, 5 of those papers are going to be incorrect.
That’s true if an effect/difference on fact exists in reality. If it doesn’t, your chance of type I error is zero, but your chance of type ii can be greater than 5% (inverse of power, which can be improved through several methods).
Something that hasn’t been said explicitly: In the OP’s example, the exact population size is irrelevant, to good approximation. That is, the significance of an additional 12 homicides is the same to many decimal places whether the underlying population is 4.5 million, 9 million, or 1 billion. As long as the population is much larger than the number of homicides itself, then it’s just the number of homicides you need to look at. (This is exactly the Poisson limit).
The last way is closest to how one should test this, though it’s not quite complete. One needs to order the possible sets of outcomes and integrate (sum) the probabilities over all worse cases. In frequentist statistics the ordering principle is a matter of choice, but using an obvious ordering for this discrete problem:
We observe 26 homicides. If each year truly has the same expectation for the number of homicides, we need to find the sum of the probabilities for all possible “splits” that are as unlikely as, or more unlikely than, the split (7, 19) we observe. There are only 27 possible splits. The ones as or more unlikely than (7, 19) have a cumulative probability of 2.9%. Or flipping the language around: There is a 97.1% chance that the two years would match better than they did if the years were truly equivalent.
It becomes an exercise for the reader to decide if this probability is significant to them.
Pasta’s test looks right to me.
However, I’m concerned about one of the OP’s earlier comments.
There is no such thing as “pure, plug-numbers-into-a-formula, statistical significance”. It is an intrinsic part of any statistical analysis that you look at the entire picture to make sure that your analysis is valid.
In this particular case, the two things that leap out at me are:
(1) Were the data cherry-picked? Pasta’s test is saying: *Under the null hypothesis of a random Poisson process, you expect to see these data, or something more extreme, less than 3% of the time. * So, where did these data come from? If, for example, there are data for 10 states over 5 years, you would expect to find at least one such occurrence. So if these data came from a motivated person who went looking for them, the result is meaningless.
You simply cannot test for significance unless you know how these particular data points were selected for analysis. Unless you know that, there is literally no such thing as a statistical test for significance.
(2) Are the model assumptions reasonable? A Poisson process is the obvious model here. But the assumption of that model is that events are independent. This is almost certainly quite badly wrong for homicides: in some cases, there will have been multiple deaths in a single incident. This kind of deviation from Poisson is known as “clumping”. It’s obvious intuitively that clumping will make a wider spread of results more likely. That’s why, when I suggested the model, I suggested it as giving a bound on significance. If, under Poisson, the result is not significant, then I think that’s probably a valid result, because any deviation from Poisson is almost certainly clumping. But if you get significance under Poisson you have more work to do to analyze the degree of clumping, and possibly find a better model to decide if the result is valid.
Green jelly beans cause acne:
And this is not just an xkcd joke.
http://www.nature.com/news/over-half-of-psychology-studies-fail-reproducibility-test-1.18248
The problem is that the multiple hypothesis testing illustrated in that xkcd cartoon is so often hiddden. Even if you’re honest, it’s so easy to fool yourself. Frequentist hypothesis testing for significance as a post hoc measure of whether we should pay attention to something is just so badly flawed that it’s completely unreliable. Unless somebody clearly states and publishes all their hypotheses beforehand, before they do any experiments, I think it’s almost meaningless.
Not in this case. I carefully limited the question to the claim of statistical significance, and even restated it to eliminate the context. Questions about the methodology or the validity of the numbers is a different part of the conversation. You are trying to take this into a realm that I specifically wanted to avoid.
Typically by looking at the credentials of the person giving you the analysis (without the raw data… .
Whats wrong in this case ? Bizarre subdivision . Taking too small a subset, for the purpose of saying there is no data. This just like saying that a bar of chocolate has 500 servicing, and there’s 0 grams of fat per serving… Snopes takes a small dataset, then says it is too small to use.
Raw data
http://www.gunpolicy.org/firearms/region/australia
Sample size obviously large enough, the change during the 1990’s is obvious ?
Well sort of …
Random breath testing, and “responsible serving of alcohol” policy, may have something to do with it, because its clear that the number of pubs has reduced … and anyway the drunk person is far less likely to have their own car nearby, and so no gun nearby.
The Australian gun control was to reduce massacre events, and that has worked.
It has also protected native wildlife, and increases the longevity of road signs.
( Maybe the ball bearing bird moved off somewhere else ? https://www.youtube.com/watch?v=Pt8-HXJ5oh8 … obviously a drinkers song 
)
Again, there is no such thing as a test for statistical significance without context.
A significance test involves (at least) all of the following steps
(1) How were the data obtained?
(2) What is the appropriate model, does the phenomenon satisfy the assumptions of the model?
(3) How do I do the math to perform a test under this model?
In your OP, you had not addressed (1) or (2).
If your OP had said, “I know that these data were randomly selected in an unbiased fashon; and I know that the phenomenon satisfies the assumptions of the Poisson model. How do I do perform a significance test under Poisson?”
Then, and only then, it would reasonable to respond with just (3).
It is not nitpicking, it is simply wrong to claim that anything represents a significance test without addressing (1) and (2). Even if you ignore (1), you need to know the appropriate model, because there are dozens of different statisticals tests applicable to different models and different kinds of data.
Riemann is on point. Taking your previous clarification:
You can’t forget that it’s about homicides. What if the thing being counted was “out of all coins returned as change over the course of a year in the U.S., how many were returned to people named Joey McJoe on Saturdays in Wisconsin?” If in two sequential years we saw 7 coins returned and 19 coins returned, is that difference significant? Even just calculating the probability of that occurring requires us to realize that change gets returned in shots with multiple coins coming at once, and that certain numbers of coins per instance are more likely than others. If instead of all coins we asked about pennies specifically, we’d get yet a different answer.
Even with a specific probability in hand under a specific model or models, it’s a separate step entirely to assess the significance of it. Here, too, you cannot divorce yourself from the actual system at hand. If the homicide problem was “After looking at 30 countries where good homicide statistics were available, we found that Australia had the most extreme deviation (p=2.9%),” then 2.9% would not be significant by any reasonable measure since there were 30 trials.
None of this is to say that you haven’t provided some context – you have. But it is incorrect to say that that context doesn’t enter when establishing the statistical significance of 7 versus 19 of anything. The context is crucial.
Sorry can’t say that, because it is too small a past incidence rate compared to population size… The appropriate way to measure it is massacres using civilian owned rifles per century or something…
All this overanalysis… Obviously when the gun related murder rate is 200 to 300 a year, we have enough data… You all fell for snopes bizarre … bizarre what ? its just bizarre talk.
Just to add: the burden of proof is obviously to show significance. This includes showing that the data sampling meets the assumptions of the statistical test.
If there is no information available about how the data were chosen, then Snopes is precisely correct to say the data are not statistically significant, and this does not require math.
Ok I read snopes carefully
Introduction… introduces some ways to make a fallacious argument from statistics. hmm that is propaganda, because it begs the question of whether they were relevant. This isn’t a page from a text book, it is meant to be a small piece which would probably be strictly on topic, OR make it clear that any irrelevant section is irrelevant.
Body… quotes some statistics, but its a real jig saw of random facts. How do they fit together ? To what falacial argument do they address ? More propaganda, by demonstrating how a single year in Victoria is too small a sample size, it was left to the reader to decide if that could be extended to the entire country over the TWO DECADES ? You all fell for basic propaganda and went on to describe Poisson…
But true , there can be other causes, because the “getting drunk at the pub every afternoon” has gone away with random breath testing and taxes, and that is good for removing the guns from drunk people, and for the populations brains .However its not clear that is so good, because it may mean the boys with pistols are less socialised - they are fearful of strangers and have less experience with conflict resolution - maybe they turn to pistols to communicate their fears… instead of talking to the other side to diffuse the situation, negotiate a peace treaty…
I didn’t read the Snopes page. I don’t even know what the political background is. I think most of the responders here (at least at the start) were just discussing how to model homicide rates, and what is required for a valid significance test.