A lot has been and is being said about Nate Silver’s accuracy, but there is one thing about his model that I don’t think I understand.
Silver didn’t make predictions per se but rather expressed the likelihood that something would happen in terms of probability, so it seems to me that for his model to be truly accurate he needed some “misses”. Even if you can say that something has a 90% chance of happening, that means that one out of ten times it won’t happen. If you say that two separate events each has a 90% chance of happening then for both of them to happen you’re looking at 90% * 90% = 81% that at least one will not happen.
If you make this kind of prediction across all 50 states then the likelihood that you will get them all right drops to .515% or about one in 200 tries (.9 ^ 50 if I remember this stuff correctly).
Even if all your state probabilities are at 99% you still only have a 60% of getting them all right. Throw in the sub 60% probabilities like Florida and your odds drop dramatically.
So it seems that as a predictive tool, Nate’s model is excellent, but as a probabilistic tool, it is much too pessimistic. Although it sounds counter intuitive, shouldn’t Nate have been hoping for some misses?
I’m no statistics expert, but in this case aren’t you’re looking at an 81% chance that both will happen, which works out to a 19% chance of at least one not happening?
Yes, you’re correct, provided the events are independent. But in this case they almost certainly are not. They are positively correlated which means the probably of both happening is between 90% (for perfect correlation) and 81% (for zero correlation). I don’t know what the correlation is.
Another thing to point out is that Silver’s model was pretty damn clear about how the vast majority of the states (i.e., the non-swing states) were going to go. By the time the election rolled around, he only had 13 states with forecast probabilities between 0.5% and 99.5%; and of these, only six (FL, NC, NH, VA, IA, CO) had forecast probabilities between 10% and 90%. You’ll notice, by the way, that Ohio is not on that list, which is why Silver thought Obama was the strong favorite in the week before the election.
FYI, if you calculate the probability of Silver getting all of the states right given the probabilities he was giving the day before the election, you get about 14%. Not impossible, but on the unlikely side.
According to Silver’s blog post a day or two before the election, almost all of the remaining chance for Romney wins was based on polls being systematically skewed. Now, it turns out that the polls were skewed, but they were skewed Republican. I haven’t done the math, but I’m pretty sure if they’d been skewed Dem as much as they were actually skewed Rep, Romney would have taken Florida at least and likely Virginia and Ohio and who knows? So I’m not sure the probability of an Obama win was understated by Silver.
Is this based on assuming independent events? Cause that’s not really plausible here.
But to answer the OP, yes. If I tell you that something will happen 95% of the time and it actually always happens, I’ve made a bad prediction. Nate got lucky this time, but he will “miss” eventually by the simple law of averages. I put “miss” in quotes because he’s not calling the states; he’s actually giving estimates of the probabilities that they’ll break each way. Not enough people understand that distinction, so it’s nice to see this question popping up.
He also missed 2 senate races. The one in North Dakota he missed by quite a bit in the sense that he assigned a high percentage to the republican, but the democrat won narrowly.
The correlation/independence factor makes sense. It’s been 40 years since my last statistics class, so **ultrafilter **would you be kind enough to give me an example of events that would be, say, moderately independent?
“Moderately independent” is technically impossible. Independent is an absolute. Things are either independent or not. “Correlated” however is a scale running from -1 to 1. So you can be moderately correlated. The simplest way to think about it, I guess is the sum of faces on dice. If you roll two dice, the numbers will be independent (correlation of zero though that’s not quite the same as independent). However, the sum of the numbers on the two dice will be correlated with the number on either one of them. The correlation is 0.707 (or more precisely 1/sqrt(2)). The correlation of sum of three dice will any one of them is 1/sqrt(3) = 0.577. Etc.*
As you add more and more dice, the relation between the sum and any one of them gets weaker and weaker. You’ll have to pick your own idea of when the correlation is moderate. The basic formula for corr[x,y] = covariance[x,y]/(stddev[x] stddev[y]).
So let x be the number on one die, z be the sum of the numbers on the other dice and y be the sum of the numbers on all the dice, y = x+z. Since the dice are independent var[y] = nvar. The covariance of x and y is cov[x, x+z] = cov[x,x] + cov[x,z] = var + 0. So corr[x,y] = var/sqrt(varnvar) = 1/sqrt(n).
Wow, OldGuy, talk about obfuscating a question with some stats jumbo that doesn’t at all help to answer the question at hand.
The simple answer to the OP’s question is that, yes, you would expect some misses…and in an ideal world, you would expect them equally in opposite directions.
But I wasn’t answering the original post. You can see from the quote, I was answering the OP’s later question for “an example” of things that are “moderately independent”.
I don’t think you’d be able to calibrate such predictions well without much more data. Silver’s early numbers had to reflect the uncertainty of events during (especially) October.
For example, some pundits seemed to think that Hurricane Sandy had a significant effect on the vote. What if a key external event been instead “pro-Romney”?
For another statistical take on various aggregators performances see here. They suggest using the “Brier’s score” to take into account the factors the op mentions.
Botom line there was that Nate was very good and some simpler models were even better.
Correlation is a measure only of the linear relationship between variables. Variables can be related without having a linear relationship. E.g., the classic example is y=x^2. y and x have a correlation of zero, but y is completely determined by x.
I shot myself in the head. Actually I appreciate OldGuy’s answer.
I guess the reason that the state numbers are correlated is because if there had been a scandal (Benghazi maybe) that hurt the President, that would drive *all *the states toward Romney, whereas a drop in the unemployment numbers would drive *all *the states towards Obama.
It’s the external events variable that adds to the uncertainty as you get further from the election. So a poll two months out that shows Obama ahead 60/40 might have a lower probability than a poll two days out that shows a 51/49 split.
Actually I appreciate OldGuy’s answer.