But isn’t that exactly what happened? Nebraska split 4-1 in 2008. A model could have easily just allocated all 5 to Nebraska. I don’t know if that is what happened, but that sounds the most plausible.
Looking at the election scorecard I linked to earlier, that’s exactly what happened. All the states were predicted correctly by Sam’s model (including Indiana). His model just awarded all 5 EVs to McCain in Nebraska, when in reality, it was 4 for McCain and 1 for Obama.
So, Sam was 50-for-50 (statewise) in 2008, missing only one EV from Nebraska. Nate missed Indiana and also that EV from Nebraska, off by 12 EV.
Mea culpa.
Actually, it’s a bit more confusing than that.
Here’s Sam’s final map. So he did miss Indiana.
But in his EV estimate he has:
So that note was added after the fact, it seems. So his original prediction was 352, and should be number we go by, I think.
However, he also adds this:
So, a little fudging going on there, it seems. In his scorecard, he calls the “364-174” prediction the official prediction, but in his state-by-state rundown, he says he missed Indiana, which makes no sense given the 364-174 EV prediction.
Oh, bloody hell.
Ezra Klein at Wonk Blog compiled all of the final predictions he could find in one thread on Monday. An interesting read…
Saw a tweet yesterday saying, “Nate Silver is drunk on the subway telling people the exact day they’ll die.”
Brad Plumer, actually.
The aggregators compared:
Wang is incompletely evaluated in that list (and I cannot understand why) and it does look like he may have done about as well.
I love 538. I love the informed analysis. But when two models both perform well the simpler of the two is to be preferred. Wang rocks (too).
There’s a bunch of them. “Drunk Nate Silver gets home 30 seconds before the Comcast installer shows up #drunknatesilver”
More scorecard by Wang. He quantifies the various models probability estimates’ predictive values by means of something called a Normalized Brier’s score.
The actual scorecard shows the PEC model’s Senate Brier score up at 0.844 and 538 down at 0.118. Not too surprising given that Wang got all ten of the closest races correct and Nate missed two.
If Nate’s a witch then what is Wang?
More seriously, going by this election, the aggregators’ performance as a group documents the superiority of meta-analysis of state polling to national polling and to fundamental factors (like that now infamous University of Colorado model). And of course to puffed up punditry. But as good as 538 is it does not appear that Nate’s secret sauce (house effect adjustments, etc.) adds anything except possibly a small amount of additional noise.
Which may be blasphemy in these parts!
DSeid, I’m pretty sure Nate’s added ingredients (especially, economic forecasts – not current data, but forecasts) improve his value early on, say, four or five months before an election. That’s when he has more of a chance of (marginally) “knowing more about who you will be three months into the future than you know yourself”.
Closer to the election, I agree, that stuff doesn’t add much, if anything. The model accounts for this, by shedding off those extra ingredients in the final weeks.
That leaves the “figuring out which pollsters are good and which are crap” thng as probably the best thing Nate does even in the last weeks. But there are others that do this well also – sounds like that Wang guy is great at it.
Synopsis of conversation with a coworker Thursday:
Coworker: sad about election
Me: I understand you’re disappointed with the outcome but you didn’t really think Romney was going to win did you?
Coworker: Yes! Yes, I did!
Me: But the polls indicated Obama was leading in the battleground states.
Coworker: Not ALL the polls.
Me: Yeah, just the honest ones.
Coworker: No, they were rigged in Democrat’s favor.
Me: So the polls that correctly predicted the outcome were RIGGED and the ones that didn’t were HONEST?
Coworker: more sadness
I’d suspect that what you are saying is correct but I do not know that. Maybe someone can go back and look at what he was predicting early on, not just this time but in 2008, and see how predictive it was compared to other models. But the value of his early posts using things like economic forecasts and so on is more explaining that there really is a political science as opposed to the pooferall of the pundits, and less having a system that outperforms.
Drunk Nate Silver.
I agree that’s the big lesson here. Another reason why his blog posts (mini-articles, really) were the best thing about Nate, IMHO. Some of them should be required reading for high schoolers.
A tile.
First of all, in the Presidential races the Brier scores are very close. Secondly, Wang may make House effect adjustments as well (not sure). What Wang does not like is throwing in economic variables in willy-nilly. Nate thinks it best to use them in June-July and phase them out as Nov 6 nears. I leaned towards Nate’s view – but Wang might be correct. The issue deserves careful analysis.
More generally, probabilistic forecasts can be evaluated by the 3 criteria proposed by Allan Murphy. The first is quality of accuracy: the Brier score gives a proxy for this. I’m wondering whether some of Nate’s forecasts were closer to 50% than warranted: it is asserted that intrade suffers from this problem, especially for low probability events. The second is consistency or honesty. Does the forecast have internal consistency. Is it the best possible forecast that could have been presented given what the forecaster knew, or was it massaged to fit the audience? cough Barone, George Will cough The weather service takes this issue very seriously and intentionally segregates its forecast professionals from those who interact with the media or even public officials. The third is economic value: does the forecast help policy makers, whether in government or business, make better decisions?
I adapted that presentation of Murphy’s work from Silver (2012) The Signal and the Noise. Heh. Now #2 at Amazon!
Geoff Berg: But the problem with people like Morris and the networks that give him airtime is not just that they’re fools or liars, it’s that the conservative movement is now dominated by people who believe that kind of obvious nonsense without thinking. Just before the election, Twitter was flooded with Fox/Limbaugh/NewsMax/WND types laughing about what a certainty it was that Barack Obama was about retire early.
Nate Silver, whose model correctly predicted both the 2008 presidential election and the 2010 midterms? Just a liberal hack in the tank for Obama. Companies like Public Policy Polling which consistently showed Obama ahead in swing states? It sometimes does polling for DailyKos and therefore must be untrustworthy.
A Fordham University study released today shows that PPP and DailyKos/SEIU/PPP were the numbers one and two most accurate polling organizations this cycle. Conservative comfort blanket Rasmussen was 24th out of 28. It’s pretty sad the way conservatives, like IIRC OMG!, thought that PPP was biased and Rasmussen was accurate. They could have looked up these groups past accuracy. Instead they used the rule of thumb “On my team/off my team” to estimate their accuracy. Those who use such methods really should be kept away from our nation’s policy levers as well as sharp objects for that matter.
Measure for Measure… you listen to Geoff Berg too???
Er, not really: I just use google news. First I’ve even heard of the guy. :o
ETA: You know what I want to see? A time series of Brier scores for the various forecasting outfits. Who cares what they thought on Nov 4? I’d like to know how their forecast accuracy tracked over time.