# Nate Silver / 538 Was Right **URL:** \ **Category:** Politics & Elections\ **Created:** [November 7, 2012, 5:43am UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181 "2012-11-07T05:43:57Z")\ **Posts on this page:** 20\ **Page:** 7
**Author:** ![drewtwo99](https://avatars.discourse-cdn.com/v4/letter/d/d07c76/32.png) [@drewtwo99](https://boards.straightdope.com/u/drewtwo99)\ **Post date:** [November 11, 2012, 5:42am UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/121 "2012-11-11T05:42:12Z")
> [@Measure\_for\_Measure](#): > > ## Er, not really: I just use google news. First I’ve even heard of the guy. :o > > ETA: You know what I want to see? A time series of Brier scores for the various forecasting outfits. Who cares what they thought on Nov 4? I’d like to know how their forecast accuracy tracked over time. Oh darn. Haha, I got so excited when I saw his name 😃 ---
**Author:** ![Measure\_for\_Measure](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/measure_for_measure/32/557_2.png) [@Measure\_for\_Measure](https://boards.straightdope.com/u/Measure_for_Measure)\ **Post date:** [November 11, 2012, 7:32pm UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/122 "2012-11-11T19:32:17Z")
> [@Measure\_for\_Measure](#): > > [Geoff Berg](http://blog.chron.com/partisangridlock/2012/11/conservatives-vs-reality/): But the problem with people like Morris and the networks that give him airtime is not just that they’re fools or liars, it’s that the conservative movement is now dominated by people who believe that kind of obvious nonsense without thinking. Just before the election, Twitter was flooded with Fox/Limbaugh/NewsMax/WND types laughing about what a certainty it was that Barack Obama was about retire early. > > Nate Silver, whose model correctly predicted both the 2008 presidential election and the 2010 midterms? Just a liberal hack in the tank for Obama. Companies like Public Policy Polling which consistently showed Obama ahead in swing states? It sometimes does polling for DailyKos and therefore must be untrustworthy. > > A Fordham University study released today shows that PPP and DailyKos/SEIU/PPP were the numbers one and two most accurate polling organizations this cycle. Conservative comfort blanket Rasmussen was 24th out of 28. I haven’t looked at the Fordham study, but Nate Silver evaluated the polling firms that he used yesterday: [Which Polls Fared Best (and Worst) in the 2012 Presidential Race - The New York Times](http://fivethirtyeight.blogs.nytimes.com/2012/11/10/which-polls-fared-best-and-worst-in-the-2012-presidential-race/) Yes, Rasmussen was near the bottom (and shockingly Gallup was last!). Public Policy Polling was below average: its average error was 2.7 while Rasmussen’s was 4.2: Gallup was a whopping 7.2. Google Consumer Surveys tied for 2nd at 1.6: we might consider that as something close to a gold standard. CNN was close at 1.9. PPP was a little closer to Google than they were to Rasmussen. I calculate the median error at 2.5: PPP was close to that but had room for improvement. (The median is 2.3 if I don’t include the two firms that Nate had dropped due to suspicions about their methodology.) PPP had a bias towards Republicans, though it was not as pronounced as Rassmussen. ---
**Author:** ![lisiate](https://avatars.discourse-cdn.com/v4/letter/l/f05b48/32.png) [@lisiate](https://boards.straightdope.com/u/lisiate)\ **Post date:** [November 11, 2012, 8:04pm UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/123 "2012-11-11T20:04:51Z")
> [@DSeid](#): > > [More scorecard](http://election.princeton.edu/2012/11/09/karl-rove-eats-a-bug/#more-8830) by Wang. He quantifies the various models probability estimates’ predictive values by means of something called a Normalized Brier’s score. > > The actual scorecard shows the PEC model’s Senate Brier score up at 0.844 and 538 down at 0.118. Not too surprising given that Wang got all ten of the closest races correct and Nate missed two. > > If Nate’s a witch then what is Wang? > > More seriously, going by this election, the aggregators’ performance as a group documents the superiority of meta-analysis of state polling to national polling and to fundamental factors (like that now infamous University of Colorado model). And of course to puffed up punditry. But as good as 538 is it does not appear that Nate’s secret sauce (house effect adjustments, etc.) adds anything except possibly a small amount of additional noise. > > Which may be blasphemy in these parts! I view Silver as very very good and Wang as slightly better, (just my opinion of course). What I find interesting about both of them is they come to poll analysis from more mathematically rigourous backgrounds - basball statistics (and economic consulting for KPMG) for Silver, Neuroscience for Wang. ANd they clean the clocks of the ‘professional’ political scientists. ---
**Author:** ![Hershele\_Ostropoler](https://avatars.discourse-cdn.com/v4/letter/h/e47c2d/32.png) [@Hershele\_Ostropoler](https://boards.straightdope.com/u/Hershele_Ostropoler)\ **Post date:** [November 12, 2012, 6:21am UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/124 "2012-11-12T06:21:26Z")
> [@Measure\_for\_Measure](#): > > PPP had a bias towards Republicans, though it was not as pronounced as Rassmussen. You mean in the mathematical sense? ---
**Author:** ![Lantern](https://avatars.discourse-cdn.com/v4/letter/l/d26b3c/32.png) [@Lantern](https://boards.straightdope.com/u/Lantern)\ **Post date:** [November 12, 2012, 6:31am UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/125 "2012-11-12T06:31:53Z")
> [@lisiate](#): > > ANd they clean the clocks of the ‘professional’ political scientists. Who are you referring to exactly? In general the fundamental models that have been developed by political scientists worked pretty well. IIRC 538 averaged their predictions and it came to around +2 for Obama. And this was months before the election. ---
**Author:** ![Measure\_for\_Measure](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/measure_for_measure/32/557_2.png) [@Measure\_for\_Measure](https://boards.straightdope.com/u/Measure_for_Measure)\ **Post date:** [November 12, 2012, 9:01am UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/126 "2012-11-12T09:01:16Z")
> [@Hershele\_Ostropoler](#): > > You mean in the mathematical sense? Well, as it happens, yeah. There are two measures. One is the average error: Nate reports the average absolute error rather than the standard distribution. The other is the bias, which consists of adding up all the errors and dividing by the number of polls - no absolute value needed. Most of the polling houses had a Republican bias, meaning that they over-estimated the vote that the Romney would ultimately secure on average. See the link: [Which Polls Fared Best (and Worst) in the 2012 Presidential Race - The New York Times](http://fivethirtyeight.blogs.nytimes.com/2012/11/10/which-polls-fared-best-and-worst-in-the-2012-presidential-race/) ## Where did the bias come from? Who knows? A likely candidate though would be the wild mismatch between Obama’s high tech GOTV effort and Romney’s disastrous ORCA project. I suspect though that there are a number of often offsetting factors to be sorted through and weighed. Here are 7 forecast models: [7 Prognosticators With Good News for Nervous Obama Fans – Mother Jones](http://www.motherjones.com/mojo/2012/11/election-predictors-obama-romney) Three are assembled by political scientists, fwiw. ---
**Author:** ![MDKSquared](https://avatars.discourse-cdn.com/v4/letter/m/b9e5f3/32.png) [@MDKSquared](https://boards.straightdope.com/u/MDKSquared)\ **Post date:** [November 12, 2012, 4:45pm UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/127 "2012-11-12T16:45:10Z")
I am surprised (although not really) that more people don’t talk about how much luck was involved in getting all 50 states correct. If you look at the ~8 states or show with non surefire probabilities: Colorado: .797 Florida: .503 Iowa: .843 Nevada: .934 New Hampshire: .846 Ohio: .906 North Carolina: .744 Virginia: .794 P(getting all 8 right) = .797\*.503\*.843\*.934\*.846\*.906\*.744\*.794 = .14 = 14% The expected number of states to get correct is about ~6.4 out of 8, with a standard deviation of ~1. Here is a screenshot of a Monte Carlo simulation (first 40 shown, out of like 14,000 because I’m lazy): [http://s8.postimage.org/ocqbor26d/excel.png](http://s8.postimage.org/ocqbor26d/excel.png) You can see how easy it is to get less than 8 correct of those non surefires. It’s impressive that Nate was 50/50, but I feel like nobody talks about how easy it would have been for him to get 48/50, especially with a coin-toss like Florida in the mix which he changed to pro-Obama at the last minute. What we should really care about here is how, even with 48/50, Obama would still have won – as well as the merits of using good data-gathering methods rather than relying on the nebulous Rove method of “The crowds are so enthusiastic, so clearly this means a landslide for Romney!”. The fact that he was 50/50 was partially due to luck and I think people are giving him too much credit. Similarly, if he had only gotten 47 or 48 out of 50, people would be go the other direction with equal magnitude: “Oh, he’s clearly no guru, some of his predictions were off!” ---
**Author:** ![Gorsnak](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/gorsnak/32/7587_2.png) [@Gorsnak](https://boards.straightdope.com/u/Gorsnak)\ **Post date:** [November 12, 2012, 5:10pm UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/128 "2012-11-12T17:10:22Z")
> [@MDKSquared](#): > > P(getting all 8 right) = > .797\*.503\*.843\*.934\*.846\*.906\*.744\*.794 = .14 = 14% You are treating these probabilities as independent, but they are not independent. ---
**Author:** ![DrDeth](https://avatars.discourse-cdn.com/v4/letter/d/b487fb/32.png) [@DrDeth](https://boards.straightdope.com/u/DrDeth)\ **Post date:** [November 12, 2012, 5:12pm UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/129 "2012-11-12T17:12:57Z")
Got a cite to his original prediction? ---
**Author:** ![bup](https://avatars.discourse-cdn.com/v4/letter/b/6bbea6/32.png) [@bup](https://boards.straightdope.com/u/bup)\ **Post date:** [November 12, 2012, 5:22pm UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/130 "2012-11-12T17:22:59Z")
> [@MDKSquared](#): > > I am surprised (although not really) that more people don’t talk about how much luck was involved in getting all 50 states correct. I mentioned it. My take is that Nate Silver is more conservative in his estimates than is warranted. I guess he knows that ‘getting a state wrong’ (which isn’t really fair, but whatever), would hurt his reputation. So he overestimates how risky the call is. His Monte-Carlo simulations, of course, are based on his stated probability for each state - if what he calls 66% is really more like 85% in his own mind, he gets a wider spread. Also, \*\*Gorsnak \*\*is right. If Obama outperforms the prediction in one state, the likelihood is higher that he will outperform the prediction in another state. ---
**Author:** ![lisiate](https://avatars.discourse-cdn.com/v4/letter/l/f05b48/32.png) [@lisiate](https://boards.straightdope.com/u/lisiate)\ **Post date:** [November 12, 2012, 9:07pm UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/131 "2012-11-12T21:07:24Z")
> [@Lantern](#): > > Who are you referring to exactly? In general the fundamental models that have been developed by political scientists worked pretty well. IIRC 538 averaged their predictions and it came to around +2 for Obama. And this was months before the election. It was more a dig at the rabid pundits than anything else. ---
**Author:** ![MDKSquared](https://avatars.discourse-cdn.com/v4/letter/m/b9e5f3/32.png) [@MDKSquared](https://boards.straightdope.com/u/MDKSquared)\ **Post date:** [November 13, 2012, 5:07pm UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/132 "2012-11-13T17:07:38Z")
> [@bup](#): > > I mentioned it. My take is that Nate Silver is more conservative in his estimates than is warranted. > > I guess he knows that ‘getting a state wrong’ (which isn’t really fair, but whatever), would hurt his reputation. So he overestimates how risky the call is. > > His Monte-Carlo simulations, of course, are based on his stated probability for each state - if what he calls 66% is really more like 85% in his own mind, he gets a wider spread. > > Also, \*\*Gorsnak \*\*is right. If Obama outperforms the prediction in one state, the likelihood is higher that he will outperform the prediction in another state. I figure the probabilities are not independent, but I felt like this should have been reflected. For instance State A: 80 people vote for Obama, 20 vote for Romney State B: 60 people vote for Obama, 40 vote for Romney Now, Nate might report these as State A = 80% for Obama, and State B = 60% for Obama, but you can use things like multivariable regression/weighted factor analysis to figure out if any correlation exists and which factors are most heavily weighted. As an example, maybe it turns out that how Obama fares in State A has a significant rippling effect that greatly pulls up the probabilities of State B despite the current-day frequency. Then again perhaps my own statistical knowledge is limited, but I felt like I had a hard time understanding what the state-by-state probabilities were supposed to reflect when it seemed clear that they aren’t all individual microcosms. ---
**Author:** ![gamerunknown](https://avatars.discourse-cdn.com/v4/letter/g/6bbea6/32.png) [@gamerunknown](https://boards.straightdope.com/u/gamerunknown)\ **Post date:** [November 14, 2012, 8:49am UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/133 "2012-11-14T08:49:03Z")
Probably deviation from the average voter per region? ---
**Author:** ![Lantern](https://avatars.discourse-cdn.com/v4/letter/l/d26b3c/32.png) [@Lantern](https://boards.straightdope.com/u/Lantern)\ **Post date:** [November 14, 2012, 11:43am UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/134 "2012-11-14T11:43:39Z")
> [@lisiate](#): > > It was more a dig at the rabid pundits than anything else. Ah OK. Let me politely suggest that you never confuse political pundits and political scientists in the presence of the latter.😃 Looks like the 538 model performed about as well as Wang’s model in the presidential race but missed pretty badly in the senate race . As I mentioned at the beginning of the thread the Montana race was an especially interesting test case because the 538 model made a different prediction to a simple polling average. Not surprisingly Nate’s posting rate has slowed down since the election but I hope he analyzes his senate predictions some time. I also agree that between two equally performing models the one with fewer parameters is generally to be preferred although it will take several more elections to judge the two approaches fairly. In any event, where Nate really adds value is the quality of his analysis. There are many competent statisticians who could build a good model to predict elections but there are very few, if any, writers who have his uncanny knack of anticipating the interesting political questions of the day, diving into the data and coming out with a lucid and detailed analysis. ---
**Author:** ![John\_Stamos\_Left\_Ear](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/john_stamos_left_ear/32/3198_2.png) [@John\_Stamos\_Left\_Ear](https://boards.straightdope.com/u/John_Stamos_Left_Ear)\ **Post date:** [November 14, 2012, 7:07pm UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/135 "2012-11-14T19:07:27Z")
> [@Measure\_for\_Measure](#): > > I haven’t looked at the Fordham study, but Nate Silver evaluated the polling firms that he used yesterday: [Which Polls Fared Best (and Worst) in the 2012 Presidential Race - The New York Times](http://fivethirtyeight.blogs.nytimes.com/2012/11/10/which-polls-fared-best-and-worst-in-the-2012-presidential-race/) > > Yes, Rasmussen was near the bottom (and shockingly Gallup was last!). Public Policy Polling was below average: its average error was 2.7 while Rasmussen’s was 4.2: Gallup was a whopping 7.2. The Gallup organization actually made a statement considering their poor showing in polling the 2012 Presidential election. As one might expect from such a venerable firm, it is apologetic and promises to reexamine their methodology so as to do a better job in the future. Ha, no. That’s actually not what they said. Actually, [they blamed Nate Silver](http://www.salon.com/2012/11/13/gallup_is_very_upset_at_nate_silver/). > [@](#): > > You don’t have to read too far between the lines of a statement from Gallup’s editor in chief, Frank Newport, published on Friday, to get that impression. > > > [@](#): > > > > But some of this will result from a variant of the venerable “law of the commons.” Individual farmers can each made a perfectly rational decision to graze their cows on the town commons. But all of these rational decisions together mean that the commons became overgrazed and, in the end, there is no grass left for any cow to graze. Many individual rational decisions can end up in a collective mess. > > > > We have a reverse law of the commons with polls. **It’s not easy nor cheap to conduct traditional random sample polls. It’s much easier, cheaper, and mostly less risky to focus on aggregating and analyzing others’ polls. Organizations that traditionally go to the expense and effort to conduct individual polls could, in theory, decide to put their efforts into aggregation and statistical analyses of other people’s polls in the next election cycle and cut out their own polling.** If many organizations make this seemingly rational decision, we could quickly be in a situation in which there are fewer and fewer polls left to aggregate and put into statistical models. Many individual rational decisions could result in a loss for the collective interest of those interested in public opinion. > > > > This will develop into a significant issue for the industry going forward. > > It is impossible to read this as anything other than an attack on Nate Silver, who is by far the most prominent aggregator and analyzer of others’ polls currently operating today. And it simply reeks of sour grapes. During the campaign year, Silver consistently pointed out that Gallup’s results were oddly inconsistent with what other pollsters were finding. And he was right — Gallup got it wrong. It is not inappropriate to point that out. But Gallup presumes too much when it effectively threatens to take its surveys home and just stop playing. > > Gallup faces a more unhappy future. If the company keeps getting the numbers wrong, the market will punish it. At that point, Gallup won’t be able to blame its sorry fate on anyone but its own incompetence. ---
**Author:** ![Gangster\_Octopus](https://avatars.discourse-cdn.com/v4/letter/g/a8b319/32.png) [@Gangster\_Octopus](https://boards.straightdope.com/u/Gangster_Octopus)\ **Post date:** [November 14, 2012, 8:00pm UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/136 "2012-11-14T20:00:01Z")
That is some serious spin. ---
**Author:** ![Marley23](https://avatars.discourse-cdn.com/v4/letter/m/45deac/32.png) [@Marley23](https://boards.straightdope.com/u/Marley23)\ **Post date:** [November 14, 2012, 8:21pm UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/137 "2012-11-14T20:21:19Z")
So Gallup’s data sucks, but they have great intangibles? ---
**Author:** ![tim314](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/tim314/32/3468_2.png) [@tim314](https://boards.straightdope.com/u/tim314)\ **Post date:** [November 14, 2012, 8:39pm UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/138 "2012-11-14T20:39:38Z")
Nobody is faulting Gallup for not performing as well as Nate Silver. At least, they shouldn’t be. As Silver himself has said many times, you can get a more accurate prediction from looking at several polls instead of just one. For one thing, it makes your sample size much larger. Where people are faulting Gallup (or should be), is for [performing worse than other individual polls.](http://fivethirtyeight.blogs.nytimes.com/2012/11/10/which-polls-fared-best-and-worst-in-the-2012-presidential-race/) Saying “it’d be easy to get a good prediction if we aggregated polls” is completely ignoring this legitimate criticism. ---
**Author:** ![DSeid](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/dseid/32/20194_2.png) [@DSeid](https://boards.straightdope.com/u/DSeid)\ **Post date:** [November 14, 2012, 8:52pm UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/139 "2012-11-14T20:52:15Z")
Gallup might stop polling? Seems like nothing lost. ---
**Author:** ![Jonathan\_Chance](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/jonathan_chance/32/701_2.png) [@Jonathan\_Chance](https://boards.straightdope.com/u/Jonathan_Chance)\ **Post date:** [November 15, 2012, 2:18am UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/140 "2012-11-15T02:18:00Z")
FYI [Deadspin Interviews Nate Silver](http://deadspin.com/5960364/skew-yourselves-nate-silver-is-here-to-answer-your-questions) > [@](#): > > I think we add value mostly by comparison because the pundits are the equivalent of the 1899 Cleveland Spiders. [Previous page](https://boards.straightdope.com/t/nate-silver-538-was-right/640181.md?page=6) [Next page](https://boards.straightdope.com/t/nate-silver-538-was-right/640181.md?page=8)