Oh darn. Haha, I got so excited when I saw his name 
I haven’t looked at the Fordham study, but Nate Silver evaluated the polling firms that he used yesterday: Which Polls Fared Best (and Worst) in the 2012 Presidential Race - The New York Times
Yes, Rasmussen was near the bottom (and shockingly Gallup was last!). Public Policy Polling was below average: its average error was 2.7 while Rasmussen’s was 4.2: Gallup was a whopping 7.2. Google Consumer Surveys tied for 2nd at 1.6: we might consider that as something close to a gold standard. CNN was close at 1.9. PPP was a little closer to Google than they were to Rasmussen. I calculate the median error at 2.5: PPP was close to that but had room for improvement. (The median is 2.3 if I don’t include the two firms that Nate had dropped due to suspicions about their methodology.)
PPP had a bias towards Republicans, though it was not as pronounced as Rassmussen.
I view Silver as very very good and Wang as slightly better, (just my opinion of course).
What I find interesting about both of them is they come to poll analysis from more mathematically rigourous backgrounds - basball statistics (and economic consulting for KPMG) for Silver, Neuroscience for Wang. ANd they clean the clocks of the ‘professional’ political scientists.
You mean in the mathematical sense?
Who are you referring to exactly? In general the fundamental models that have been developed by political scientists worked pretty well. IIRC 538 averaged their predictions and it came to around +2 for Obama. And this was months before the election.
Well, as it happens, yeah.
There are two measures. One is the average error: Nate reports the average absolute error rather than the standard distribution. The other is the bias, which consists of adding up all the errors and dividing by the number of polls - no absolute value needed. Most of the polling houses had a Republican bias, meaning that they over-estimated the vote that the Romney would ultimately secure on average. See the link: Which Polls Fared Best (and Worst) in the 2012 Presidential Race - The New York Times
Where did the bias come from? Who knows? A likely candidate though would be the wild mismatch between Obama’s high tech GOTV effort and Romney’s disastrous ORCA project. I suspect though that there are a number of often offsetting factors to be sorted through and weighed.
Here are 7 forecast models: 7 Prognosticators With Good News for Nervous Obama Fans – Mother Jones Three are assembled by political scientists, fwiw.
I am surprised (although not really) that more people don’t talk about how much luck was involved in getting all 50 states correct.
If you look at the ~8 states or show with non surefire probabilities:
Colorado: .797
Florida: .503
Iowa: .843
Nevada: .934
New Hampshire: .846
Ohio: .906
North Carolina: .744
Virginia: .794
P(getting all 8 right) =
.797*.503*.843*.934*.846*.906*.744*.794 = .14 = 14%
The expected number of states to get correct is about ~6.4 out of 8, with a standard deviation of ~1.
Here is a screenshot of a Monte Carlo simulation (first 40 shown, out of like 14,000 because I’m lazy): http://s8.postimage.org/ocqbor26d/excel.png
You can see how easy it is to get less than 8 correct of those non surefires.
It’s impressive that Nate was 50/50, but I feel like nobody talks about how easy it would have been for him to get 48/50, especially with a coin-toss like Florida in the mix which he changed to pro-Obama at the last minute.
What we should really care about here is how, even with 48/50, Obama would still have won – as well as the merits of using good data-gathering methods rather than relying on the nebulous Rove method of “The crowds are so enthusiastic, so clearly this means a landslide for Romney!”. The fact that he was 50/50 was partially due to luck and I think people are giving him too much credit. Similarly, if he had only gotten 47 or 48 out of 50, people would be go the other direction with equal magnitude: “Oh, he’s clearly no guru, some of his predictions were off!”
You are treating these probabilities as independent, but they are not independent.
Got a cite to his original prediction?
I mentioned it. My take is that Nate Silver is more conservative in his estimates than is warranted.
I guess he knows that ‘getting a state wrong’ (which isn’t really fair, but whatever), would hurt his reputation. So he overestimates how risky the call is.
His Monte-Carlo simulations, of course, are based on his stated probability for each state - if what he calls 66% is really more like 85% in his own mind, he gets a wider spread.
Also, **Gorsnak **is right. If Obama outperforms the prediction in one state, the likelihood is higher that he will outperform the prediction in another state.
It was more a dig at the rabid pundits than anything else.
I figure the probabilities are not independent, but I felt like this should have been reflected.
For instance
State A: 80 people vote for Obama, 20 vote for Romney
State B: 60 people vote for Obama, 40 vote for Romney
Now, Nate might report these as State A = 80% for Obama, and State B = 60% for Obama, but you can use things like multivariable regression/weighted factor analysis to figure out if any correlation exists and which factors are most heavily weighted.
As an example, maybe it turns out that how Obama fares in State A has a significant rippling effect that greatly pulls up the probabilities of State B despite the current-day frequency.
Then again perhaps my own statistical knowledge is limited, but I felt like I had a hard time understanding what the state-by-state probabilities were supposed to reflect when it seemed clear that they aren’t all individual microcosms.
Probably deviation from the average voter per region?
Ah OK. Let me politely suggest that you never confuse political pundits and political scientists in the presence of the latter.
Looks like the 538 model performed about as well as Wang’s model in the presidential race but missed pretty badly in the senate race . As I mentioned at the beginning of the thread the Montana race was an especially interesting test case because the 538 model made a different prediction to a simple polling average. Not surprisingly Nate’s posting rate has slowed down since the election but I hope he analyzes his senate predictions some time. I also agree that between two equally performing models the one with fewer parameters is generally to be preferred although it will take several more elections to judge the two approaches fairly.
In any event, where Nate really adds value is the quality of his analysis. There are many competent statisticians who could build a good model to predict elections but there are very few, if any, writers who have his uncanny knack of anticipating the interesting political questions of the day, diving into the data and coming out with a lucid and detailed analysis.
The Gallup organization actually made a statement considering their poor showing in polling the 2012 Presidential election. As one might expect from such a venerable firm, it is apologetic and promises to reexamine their methodology so as to do a better job in the future.
Ha, no. That’s actually not what they said. Actually, they blamed Nate Silver.
That is some serious spin.
So Gallup’s data sucks, but they have great intangibles?
Nobody is faulting Gallup for not performing as well as Nate Silver. At least, they shouldn’t be. As Silver himself has said many times, you can get a more accurate prediction from looking at several polls instead of just one. For one thing, it makes your sample size much larger.
Where people are faulting Gallup (or should be), is for performing worse than other individual polls. Saying “it’d be easy to get a good prediction if we aggregated polls” is completely ignoring this legitimate criticism.
Gallup might stop polling? Seems like nothing lost.