End members are great, aren’t they? Never mind what happens if the results somewhere between 300 and 900. 
Regardless, ultrafilter’s got a better handle on the terminology than I do, and “forecast” is definitely a more accurate term.
Part of the problem is that the methodology is laid out in a series of pay-walled posts and on his old blog. So people like OMG can say things like he does here and it takes half-an-hour to point to the specific part of the methodology rebutting his statement (in this case, the post describing how the weighting is not “arbitrary”). It makes sense as a business model for nytimes.com, but it does help GOP partisans make silly methodological arguments because it takes a lot of time to comprehensively rebut them.
A plug for some very cogent commentary by my buddy Jeff at his blog Simply Statistics:
On weather forecasts, Nate Silver, and the politicization of statistical illiteracy
Actually, anyone can check out the methodology. This is the Seante Methodology, but the Presidential is pretty much the same. It is very transparent.
There are a couple extra wrinkles in the presidential model due to the fact that the election occurs across multiple states, but yeah, it’s pretty much the same.
Also, anybody who’s interested in this topic should read Nate’s book.
There are some significant differences and additional levels with the presidential modeling, actually.
True there is some cross correlation and some integration of national polling, and of course the step of aggregating the state results to get a national result. But in many ways it is basically the same, taking polls, weight them not based on his ideas about what is right,but based on objective measures, throwing in other variables about economics, demographics etc that have proven historically to be relevant (again not based on his beliefs about what should or shouldn’t be in there) and gets his forecast. It is very rigorous. but it is also not a crystal ball.
I agree that they are fundamentally similar. But a lot of the misguided critiques concern issues not covered in that methodology link, unfortunately.
My understanding of where Silver is getting the popular vote number is by taking the numbers state-by-state AND taking the numbers from the national polls and combining them. Right now, the state polls and the national polls don’t match. (They might if you dropped Gallup out of the national polls since I don’t think Gallup does state polling. I’m not sure about this.)
As we all, I hope, know, national polls aren’t completely meaningless, but they are not a good proxy for state polls when the race is close. If the race were not close, the national polls would make a good proxy, since historically there haven’t been any weird results in a presidential election when there’s a healthy margin between candidates.
I could be wrong, but I also thought that he tended to drop more and more of the variables as the election nears so that his forecast percentage went up? Of course, I might be thinking solely of 2008 in which there wasn’t much doubt about what was going to happen with the electoral college.
He does drop the influence of things like the effect of the economy. More importantly, it’s not that he drops variables, he reduces variability which is kind of like margin of error.
He definitely drops economic factors as the election nears, because he figures that will be fully reflected in people’s voting intentions by that point. He mentioned that just the other day.
Thanks!
538’s adjustments aren’t particular mysterious or controversial and have been discussed from time to time on the blog:
Off the top of my head they include things like:
a) economic data including state-by-state data
b) the quality of the poll includng the sample size and whether the poll uses mobile phones
c) the bias of the poll in either direction
d) well-known effects like the convention bounce at the appropriate time
e) “state fundamentals” which I think includes variables like past state results relative to the popular vote
Furthermore I don’t think these adjustments make that much of a difference. Just take a simple average of the last 5 or 10 polls in each state and the broad picture is mostly the same as 538. People who have a serious problem with the 538 model also have a serious problem with the state polls in general.
There is a Dilbert for that.
It’s a political thread. We need false balance. All sides are always equally right and wrong, remember.
Was Silver’s track record significantly worse in 2010 than 2008?
Well, no. They could both be right, sort of.
538 calls for Obama 295, Romney 243 as of right now. But Silver’s not “predicting” 295 EVs; that’s the average of all the model elections he ran. He is not actually saying Obama will win precisely 295. As a matter of fact, that specific outcome is amazingly unlikely, because there are very few realistic outcomes that yield precisely 295. (I can’t come up with one; I can come up with 294, though.) Silver’s model actually has the single likeliest outcome of the election being Obama 332, Romney 206 (I think; it’s hard to read the little lines) which occurs about 13% of the time he ran the last set of simulations. 295 is the average of ALL his simulations. Although 338 is the mode, it’s nowhere close to the mean; it’s just the single likeliest EV total in the general category of “Romney does suibstantially worse than it currently appears he will.”
You have to bear in mind that Silver’s model could be very, very accurate and miss twenty states. If Silver projects that Virginia will go 50.2-49.8 for Romney, but it goes 50.1-49.9 for Obama, that’s not a bad job. In fact it’s great; Silver’s average percentage miss in 2008 was 1.1 percent towards McCain. (He predicted Obama would carry the nation by 6.1 percent; he carried it by 7.2.) But if he misses by a tiny fraction in a state that’s right on the fence (as he did with Indiana in 2008 - the only state he missed, plus the single vote for NE-2) he misses the EV number even though his projection might have been really, really close.
So in fact it is theoretically possible Romney could get 330 EVs and Silver’s November 5 analysis could still be damn close, if a lot of states are right on the fence.
No. He was pretty damn good.
Silver correctly got all the Senate races but three. Two were predicted to be extremely close and were, but just on the other side of the line. The only one he really missed as Harry Reid being re-elected over Sharron Angle, he predicted Angle would win.
He predicted a net Republican gain of 54-55 seats in the House; they gained 63. That’s an okay prediction. It’s obviously harder to get the House right.
In gubernatorial races, Silver was 36-for-37. The only miss was, again, razor tight.
Also the significance of predicting something with a 95% (as opposed to merely 75%) probablity of being correct is overstated. 95% doesn’t mean “slam dunk,” it means “19 out of 20,” and is cause for some mild nervousness on he part of those who want the 95% chance to come true. I wouldn’t feel confident in my bets unless they’re somewhere around 98 or 99%.
Setting aside that he was pretty good, I wouldn’t want to compare results between a presidential election and midterms anyway. How much polling data is there for any given House race versus this year’s presidential election polling in Virginia or Nevada?
Just to be clear, I wasn’t slamming Silver. I also didn’t mean to bring up “prediction” v “forecast” I was just wondering how someone could ever be wrong talking about probabilities. Yes, you could run the scenario a million times, but since we only have elections every 2 years, we can’t do that.
Some have mentioned 35 of 37 or some such number. If anything that shows his confidence level was too LOW. It looks more like he drew an inside straight than anything about his model.