I don’t think there is remotely enough data to model the likelihood of a result-changing event taking place, and I don’t think NS is doing that.
I imagine he would acknowledge this, if you asked him. As the article notes, he tends to hedge in his actual discussions.
I think part of the disconnect results from his putting these rather precise probabilities around his numbers, e.g. 74.3% chance of winning. There is no way he has a model remotely robust enough to give a probability to the third digit. There’s a general rule in math-related fields that you try to avoid giving the impression that you know something with more precision than you actually do, and NS might be running a bit short in this regard.
But I think there’s a disconnect between what NS sees himself as doing and what people understand him to be doing. He constructed a model that spits out these probabilities, and to the extent that the model and the assumptions that underly it are valid then these numbers are accurate. But the model itself and the assumptions used in it themselves have a variance, which lessens the overall probability.
ISTM that NS is careful to say that “the model is showing X% probability that Obama wins” versus “there is X% probability that Obama wins”, and all he is saying is that these are the numbers if you accept the model, as above. NS obviously believes the model to be a very good one, but I don’t think he thinks his model can give absolute probabilities as precise as people assume.
They’re just mad he’s predicting an Obama victory.
What’s interesting is that there are a half-dozen other comprehensive models out there, at least one of which is run by an avowed Republican. All of them that incorporate polling are predicting an Obama victory.
But they don’t take as much shit as Silver, I guess because 538 is the talk of the town and the rest aren’t. But it really puts the silliness of their criticisms in perspective.
I agree with this. Most of the heat NS takes is from the right, because he is predicting an Obama victory.
ISTM that although he himself is a liberal, he strives to be impartial. To the extent that there is some subjectivity in the assumptions and methodology used I would not be surprised to see it have a slight influence. But I also wouldn’t be surprised to see it not being an influence.
In any event, the vast majority of people interested in politics to the point of putting out a model are going to have some viewpoint of their own. So the issue is unavoidable in any event.
It’s silly to argue about it from a partisan perspective, though. If NS is off it’s not by a whole lot, and it’s silly to argue that it’s a true tossup (or that Romney is favorite), as some conservative critics and pundits seem to be arguing.
538, in part, uses historical elections to predict a candidate’s probability and margin of victory given what the polls show for that candidate. The way they measure these black swan events is by including races where a candidate had such an event impact their candidacy.
Now, these tail events are presumably very uncommon, and their is likely a large variance in the frequency of such incidents, so there is a large uncertainty in predicting if such an event will occur. But, the 538 model does implicitly include the possibility of Surprise events.
Also his probability of victory is really just an average rate of the simulations that he runs on a given day. He’s been fairly clear on this in the past.
I’ve not seen him claim that. In addition, every “Event” is fundamentally different than any other event.
Plus, in this respect presidential elections are very different than other elections, since there is much more attention paid to them, and there are too few presidential elections to predict the impact of unusual events, especially as the impact of an event is different for a close election as for a runaway one.
I think it’s a weighted average. At least it should be. But I don’t know why this would be significant in terms of what we’re discussing.
I think it’s pretty clear that the model assigns weight to the polling and economic variables based on the historical predictive value of these factors.
Every event may be different, but their impacts are able to be quantified. If out of 100 candidates with a 2.5% lead one week before election day, 90 of them are elected, then that implies that 10% of the time something happens to make the candidate lose. It could be that the polls are off, a hurricane came by to disrupt turnout, or it could be that a live boy showed up in the candidate’s bed. It doesn’t matter what actually happened, only that something did happen, and that we can measure the impact it had on the candidate’s vote share.
Nate Silver calibrates his model based on the outcomes of prior elections. This includes elections where these events take place, and thus he includes a rough measurement of something derailing Obama’s candidacy. You are correct in that presidential elections do not have a large sample size to draw from, being once every four years, but using lesser-office elections helps as a proxy.
This is all very rough, as I’m trying to emphasize, so it’s likely that the 538 model is conservative in its estimation. Like in the 90/100 example above, the model may reduce that by 10% of the failure rate to 88/100, just because it is so difficult to model if such an incident will occur. I don’t know specifically if the 538 model does this, however.
It makes every difference, because the types of events are both individually rate and unrelated. It’s like pricing an insurance policy that covers lightning strikes, baboon attacks, laughing sickness, and meteor falls, based on a fairly small sample size.
I think the small sample size of incidents of baboon attacks is an excellent reason not to worry too much about including it in a prediction or forecast, though if Nate Silver wants to say, “Oh, all of these forecasts are void if there is a massive baboon attack,” I’d be okay with that, too.
Yes, with one week until the election, something unexpected could still happen. I’d prefer we stick within the realm of the plausible-to-likely.
It was an illustration of the general concept, that being that it can be difficult to use experience to calculate the combined likelihood of any number of rare and erratic incidents.
In that case you should be fine with a margin.
The question we are discussing is not whether - at this point - Obama or Romney are more likely to win. The question is how much confidence we can have in Nate Silver’s assessment that Obama’s chances are currently 78%, versus 68% or 88% (for example).
You try to estimate the chance of something crazy happening by looking at the rate of crazy shit happening in prior elections. Baboons, lightning strikes, and armageddon are assumed to happen as aggregated crazy shit. It doesn’t matter from a statistical perspective what the difference between them is.
His contention is useful in a statistical sense I think. It is possible for a poll to be within the range of the others and still perform drably (if all other polls tend to be off by a point on average and for this particular poll to slightly emphasise that). If they suddenly adopt stricter protocols and make a prediction out of line of the other polls, in the favour of a particular candidate, they could be an anomaly and outperform the other polls. However, in such an scenario, it’d also be possible for a poll to actually do better than average within the average range (in other words, be half a point off on average) and yet perform abysmally when it veers outside the range of the other polls. Say the average poll is 47:47 with a 3 point spread. Sprint polling could typically give 48% for the eventual victor, but their more common anomaly would be to give 70% to the eventual loser. Rasmetzen could give 48% to the eventual loser on average and 52% to the victor when the other polling places were giving them 47%.