# Nate Silver / 538 Was Right **URL:** \ **Category:** Politics & Elections\ **Created:** [November 7, 2012, 5:43am UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181 "2012-11-07T05:43:57Z")\ **Posts on this page:** 20\ **Page:** 6
**Author:** ![pulykamell](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/pulykamell/32/3166_2.png) [@pulykamell](https://boards.straightdope.com/u/pulykamell)\ **Post date:** [November 8, 2012, 3:14pm UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/101 "2012-11-08T15:14:38Z")
> [@Really\_Not\_All\_That\_Bright](#): > > Being off by one EV means you did poorly, rather than well. Being off by one means you got _two_ states wrong that happen to be one EV apart (unless you somehow failed to predict how Nebraska would go). But isn’t that exactly what happened? Nebraska split 4-1 in 2008. A model could have easily just allocated all 5 to Nebraska. I don’t know if that is what happened, but that sounds the most plausible. ---
**Author:** ![pulykamell](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/pulykamell/32/3166_2.png) [@pulykamell](https://boards.straightdope.com/u/pulykamell)\ **Post date:** [November 8, 2012, 3:17pm UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/102 "2012-11-08T15:17:10Z")
Looking at the election scorecard I linked to earlier, that’s exactly what happened. All the states were predicted correctly by Sam’s model (including Indiana). His model just awarded all 5 EVs to McCain in Nebraska, when in reality, it was 4 for McCain and 1 for Obama. So, Sam was 50-for-50 (statewise) in 2008, missing only one EV from Nebraska. Nate missed Indiana and also that EV from Nebraska, off by 12 EV. ---
**Author:** ![Really\_Not\_All\_That\_Bright](https://avatars.discourse-cdn.com/v4/letter/r/e8c25b/32.png) [@Really\_Not\_All\_That\_Bright](https://boards.straightdope.com/u/Really_Not_All_That_Bright)\ **Post date:** [November 8, 2012, 3:21pm UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/103 "2012-11-08T15:21:45Z")
Mea culpa. ---
**Author:** ![pulykamell](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/pulykamell/32/3166_2.png) [@pulykamell](https://boards.straightdope.com/u/pulykamell)\ **Post date:** [November 8, 2012, 3:30pm UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/104 "2012-11-08T15:30:45Z")
Actually, it’s a bit more confusing than that. [Here’s Sam’s final map](http://election.princeton.edu/2008/11/04/final-predictions-for-2008/#more-2288). So he did miss Indiana. But in his EV estimate he has: > [@](#): > > The final polling snapshot is Obama 352 EV, McCain 186 EV. The confidence bands are 68% [337,367] Obama EV, 95% [316,378] Obama EV. (Note added, Nov. 24: The snapshot was not stable. Variance minimization identifies longer periods for averaging, which give a result of 364 EV.) So that note was added after the fact, it seems. So his original prediction was 352, and should be number we go by, I think. However, he also adds this: > [@](#): > > Bias analysis: I expect cell phone users missing from landline surveys to give Obama a 1% boost. Based on the bias adjustment I gave you earlier, this scenario brings the median to a level that is still within the 68% confidence band above. I make my own personal (not official!) prediction: Obama 364 EV, McCain 174 EV. So, a little fudging going on there, it seems. In his scorecard, he calls the “364-174” prediction the official prediction, but in his state-by-state rundown, he says he missed Indiana, which makes no sense given the 364-174 EV prediction. ---
**Author:** ![Really\_Not\_All\_That\_Bright](https://avatars.discourse-cdn.com/v4/letter/r/e8c25b/32.png) [@Really\_Not\_All\_That\_Bright](https://boards.straightdope.com/u/Really_Not_All_That_Bright)\ **Post date:** [November 8, 2012, 3:34pm UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/105 "2012-11-08T15:34:51Z")
Oh, bloody hell. ---
**Author:** ![John\_Stamos\_Left\_Ear](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/john_stamos_left_ear/32/3198_2.png) [@John\_Stamos\_Left\_Ear](https://boards.straightdope.com/u/John_Stamos_Left_Ear)\ **Post date:** [November 8, 2012, 4:14pm UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/106 "2012-11-08T16:14:14Z")
[Ezra Klein at Wonk Blog](http://www.washingtonpost.com/blogs/ezra-klein/wp/2012/11/05/pundit-accountability-the-official-2012-election-prediction-thread/) compiled all of the final predictions he could find in one thread on Monday. An interesting read… ---
**Author:** ![DigitalC](https://avatars.discourse-cdn.com/v4/letter/d/ebca7d/32.png) [@DigitalC](https://boards.straightdope.com/u/DigitalC)\ **Post date:** [November 8, 2012, 6:28pm UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/107 "2012-11-08T18:28:05Z")
Saw a tweet yesterday saying, “Nate Silver is drunk on the subway telling people the exact day they’ll die.” ---
**Author:** ![Tom\_Scud](https://avatars.discourse-cdn.com/v4/letter/t/f08c70/32.png) [@Tom\_Scud](https://boards.straightdope.com/u/Tom_Scud)\ **Post date:** [November 8, 2012, 7:07pm UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/108 "2012-11-08T19:07:02Z")
> [@John\_Stamos\_Left\_Ear](#): > > [Ezra Klein at Wonk Blog](http://www.washingtonpost.com/blogs/ezra-klein/wp/2012/11/05/pundit-accountability-the-official-2012-election-prediction-thread/) compiled all of the final predictions he could find in one thread on Monday. An interesting read… Brad Plumer, actually. ---
**Author:** ![DSeid](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/dseid/32/20194_2.png) [@DSeid](https://boards.straightdope.com/u/DSeid)\ **Post date:** [November 8, 2012, 10:56pm UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/109 "2012-11-08T22:56:57Z")
The [aggregators compared](http://news.cnet.com/8301-13578_3-57546778-38/among-the-top-election-quants-nate-silver-reigns-supreme/): > [@](#): > > In addition to picking the winner in all 50 states – besting his 49 out of 50 slate in 2008 – Silver was also the closest among the aggregators to picking the two candidates’ popular vote percentages. All told, he missed Obama’s total of 50.8 percent by just four-tenths of a percentage point (50.4) and Romney’s 48 percent by just three-tenths of a point (48.3) for an average miss of just 0.35 percentage points. HuffPo Pollster and RealClearPolitics tied for second with an average miss of 0.85 points. > In preparing to make these comparisons, CNET surveyed 11 swing states. In the end, Silver was closest to the final margins among the candidates in seven of them and also had the best overall record, missing by an average of just 1.46 points in the 11 states. TPM PollTracker was second with the closest predicted margins in three states, and the second-best average margin, 1.80 points. > It is worth noting that while Silver’s final pre-election calculations showed a tie vote in Florida, he still predicted a 50.3 chance that the president would prevail in the Sunshine State. > The performances by Silver and his fellow polling aggregators should be sounding alarm bells in the halls of long-venerated pollsters like Gallup – which, by the way, predicted that Romney would win the national popular vote by a point. Wang is incompletely evaluated in that list (and I cannot understand why) and it does look like he may have done [about as well](http://election.princeton.edu/). > [@](#): > > The popular vote is Obama 51.1% to Romney 48.9%. Again, this exactly matches my prediction, which was state polls with a little Bayesian help from national polls. … Senate: Of the closest races, election returns match polling medians in 10 out of 10. Of particular note is the North Dakota race, where Heitkamp (D) leads Berg (R), 50.5-49.5%. That’s one where I had Heitkamp based on polls, and Nate Silver had Berg based on polls plus other factors. We are also waiting on the Montana race, where Tester (D) leads Rehberg, 49-45%. The upper chamber appears headed right for the median that we predicted, 55 D/I to 45 R. … I love 538. I love the informed analysis. But when two models both perform well the simpler of the two is to be preferred. Wang rocks (too). ---
**Author:** ![TokyoBayer](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/tokyobayer/32/13989_2.png) [@TokyoBayer](https://boards.straightdope.com/u/TokyoBayer)\ **Post date:** [November 9, 2012, 1:40am UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/110 "2012-11-09T01:40:33Z")
> [@DigitalC](#): > > Saw a tweet yesterday saying, “Nate Silver is drunk on the subway telling people the exact day they’ll die.” There’s a bunch of them. “Drunk Nate Silver gets home 30 seconds before the Comcast installer shows up #drunknatesilver” ---
**Author:** ![DSeid](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/dseid/32/20194_2.png) [@DSeid](https://boards.straightdope.com/u/DSeid)\ **Post date:** [November 10, 2012, 6:29am UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/111 "2012-11-10T06:29:51Z")
[More scorecard](http://election.princeton.edu/2012/11/09/karl-rove-eats-a-bug/#more-8830) by Wang. He quantifies the various models probability estimates’ predictive values by means of something called a Normalized Brier’s score. > [@](#): > > This score is the average of the squared deviations from a perfect prediction. For example, if Obama won a race that we said was 90% probable, that’s a score of (1.0-0.9)^2 = 0.01. If we were only 70% sure, the score is (1.0-0.7)^2 = 0.09. The average score for all 51 races is the Brier score. The Brier score rewards being correct – and rewards high confidence. … > > … We appear to be slightly better than our very able colleagues. The additional factors used by the FiveThirtyEight model include national polls and maybe some other parameters. It seems that these parameters did not help. > > A more interesting case is the Senate … In this case, additional factors used by FiveThirtyEight – “fundamentals” – may have actively hurt the prediction. This suggests that fundamentals are helpful mainly when polls are not available. Hmmm…should Nate eat part of a bug? The actual scorecard shows the PEC model’s Senate Brier score up at 0.844 and 538 down at 0.118. Not too surprising given that Wang got all ten of the closest races correct and Nate missed two. If Nate’s a witch then what is Wang? More seriously, going by this election, the aggregators’ performance as a group documents the superiority of meta-analysis of state polling to national polling and to fundamental factors (like that now infamous University of Colorado model). And of course to puffed up punditry. But as good as 538 is it does not appear that Nate’s secret sauce (house effect adjustments, etc.) adds anything except possibly a small amount of additional noise. Which may be blasphemy in these parts! ---
**Author:** ![JKellyMap](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/jkellymap/32/15880_2.png) [@JKellyMap](https://boards.straightdope.com/u/JKellyMap)\ **Post date:** [November 10, 2012, 12:07pm UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/112 "2012-11-10T12:07:19Z")
DSeid, I’m pretty sure Nate’s added ingredients (especially, economic forecasts – not current data, but forecasts) improve his value early on, say, four or five months before an election. That’s when he has more of a chance of (marginally) “knowing more about who you will be three months into the future than you know yourself”. Closer to the election, I agree, that stuff doesn’t add much, if anything. The model accounts for this, by shedding off those extra ingredients in the final weeks. That leaves the “figuring out which pollsters are good and which are crap” thng as probably the best thing Nate does even in the last weeks. But there are others that do this well also – sounds like that Wang guy is great at it. ---
**Author:** ![Batfish](https://avatars.discourse-cdn.com/v4/letter/b/a3d4f5/32.png) [@Batfish](https://boards.straightdope.com/u/Batfish)\ **Post date:** [November 10, 2012, 12:41pm UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/113 "2012-11-10T12:41:39Z")
Synopsis of conversation with a coworker Thursday: Coworker: _sad about election_ Me: I understand you’re disappointed with the outcome but you didn’t really think Romney was going to win did you? Coworker: Yes! Yes, I did! Me: But the polls indicated Obama was leading in the battleground states. Coworker: Not ALL the polls. Me: Yeah, just the honest ones. Coworker: No, they were rigged in Democrat’s favor. Me: So the polls that correctly predicted the outcome were RIGGED and the ones that didn’t were HONEST? Coworker: _more sadness_ ---
**Author:** ![DSeid](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/dseid/32/20194_2.png) [@DSeid](https://boards.straightdope.com/u/DSeid)\ **Post date:** [November 10, 2012, 1:20pm UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/114 "2012-11-10T13:20:47Z")
> [@JKellyMap](#): > > DSeid, I’m pretty sure Nate’s added ingredients (especially, economic forecasts – not current data, but forecasts) improve his value early on… I’d suspect that what you are saying is correct but I do not know that. Maybe someone can go back and look at what he was predicting early on, not just this time but in 2008, and see how predictive it was compared to other models. But the value of his early posts using things like economic forecasts and so on is more explaining that there really is a political _science_ as opposed to the pooferall of the pundits, and less having a system that outperforms. ---
**Author:** ![BigAppleBucky](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/bigapplebucky/32/178_2.png) [@BigAppleBucky](https://boards.straightdope.com/u/BigAppleBucky)\ **Post date:** [November 10, 2012, 1:52pm UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/115 "2012-11-10T13:52:44Z")
> [@pulykamell](#): > > Has anyone posted [the Nate Silver parody](http://www.washingtonpost.com/blogs/election-2012/wp/2012/11/07/nate-silver-twitter-parody-has-a-psychedelic-election-night/) twitter feed. It’s awesome stuff. > > A sample: Drunk Nate Silver. > **['Drunk Nate Silver': The best tweets from the funniest post-election meme](https://theweek.com/articles/470529/drunk-nate-silver-best-tweets-from-funniest-postelection-meme)** > > The New York Times stats wizard was spot-on in his election forecasts, defying his many critics. And all the thanks he got was Twitter immortality > [@](#): > > “Drunk Nate Silver, updating the Wikipedia page of the 8 year-old who’ll win the 2052 election” > – Jack Bohrer > > “Drunk Nate Silver crashing weddings, yelling out the number of years until divorce” > – Ben Keegan > > “Drunk Nate Silver knows exactly how long Drunk Nate Silver meme will be funny” > – Kaitlyn Dowling ---
**Author:** ![JKellyMap](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/jkellymap/32/15880_2.png) [@JKellyMap](https://boards.straightdope.com/u/JKellyMap)\ **Post date:** [November 10, 2012, 1:53pm UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/116 "2012-11-10T13:53:28Z")
> [@DSeid](#): > > …But the value of his early posts using things like economic forecasts and so on is more explaining that there really is a political _science_ as opposed to the pooferall of the pundits, and less having a system that outperforms. I agree that’s the big lesson here. Another reason why his blog posts (mini-articles, really) were the best thing about Nate, IMHO. Some of them should be required reading for high schoolers. ---
**Author:** ![gamerunknown](https://avatars.discourse-cdn.com/v4/letter/g/6bbea6/32.png) [@gamerunknown](https://boards.straightdope.com/u/gamerunknown)\ **Post date:** [November 10, 2012, 2:15pm UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/117 "2012-11-10T14:15:27Z")
> [@DSeid](#): > > If Nate’s a witch then what is Wang? A tile. ---
**Author:** ![Measure\_for\_Measure](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/measure_for_measure/32/557_2.png) [@Measure\_for\_Measure](https://boards.straightdope.com/u/Measure_for_Measure)\ **Post date:** [November 11, 2012, 5:11am UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/118 "2012-11-11T05:11:44Z")
> [@DSeid](#): > > I love 538. I love the informed analysis. But when two models both perform well the simpler of the two is to be preferred. Wang rocks (too). > [@DSeid](#): > > [More scorecard](http://election.princeton.edu/2012/11/09/karl-rove-eats-a-bug/#more-8830) by Wang. He quantifies the various models probability estimates’ predictive values by means of something called a Normalized Brier’s score. > > The actual scorecard shows the PEC model’s Senate Brier score up at 0.844 and 538 down at 0.118. Not too surprising given that Wang got all ten of the closest races correct and Nate missed two. > > If Nate’s a witch then what is Wang? > > More seriously, going by this election, the aggregators’ performance as a group documents the superiority of meta-analysis of state polling to national polling and to fundamental factors (like that now infamous University of Colorado model). And of course to puffed up punditry. But as good as 538 is it does not appear that Nate’s secret sauce (house effect adjustments, etc.) adds anything except possibly a small amount of additional noise. > > Which may be blasphemy in these parts! First of all, in the Presidential races the Brier scores are very close. Secondly, Wang may make House effect adjustments as well (not sure). What Wang does _not_ like is throwing in economic variables in willy-nilly. Nate thinks it best to use them in June-July and phase them out as Nov 6 nears. I leaned towards Nate’s view – but Wang might be correct. The issue deserves careful analysis. More generally, probabilistic forecasts can be evaluated by the [3 criteria](http://www.cawcr.gov.au/projects/verification/) proposed by Allan Murphy. The first is quality of accuracy: the Brier score gives a proxy for this. I’m wondering whether some of Nate’s forecasts were closer to 50% than warranted: it is asserted that intrade suffers from this problem, especially for low probability events. The second is consistency or honesty. Does the forecast have internal consistency. Is it the _best_ possible forecast that could have been presented given what the forecaster knew, or was it massaged to fit the audience? _cough_ Barone, George Will _cough_ The weather service takes this issue very seriously and intentionally segregates its forecast professionals from those who interact with the media or even public officials. The third is economic value: does the forecast help policy makers, whether in government or business, make better decisions? I adapted that presentation of Murphy’s work from Silver (2012) _The Signal and the Noise_. Heh. Now [#2 at Amazon!](http://www.amazon.com/The-Signal-Noise-Predictions-Fail-but/dp/159420411X/ref=sr_1_1?ie=UTF8&qid=1352610176&sr=8-1&keywords=signal+noise) > [@Batfish](#): > > Synopsis of conversation with a coworker Thursday: > > Coworker: _sad about election_ > Me: I understand you’re disappointed with the outcome but you didn’t really think Romney was going to win did you? > Coworker: Yes! Yes, I did! > Me: But the polls indicated Obama was leading in the battleground states. > Coworker: Not ALL the polls. > Me: Yeah, just the honest ones. > Coworker: No, they were rigged in Democrat’s favor. > Me: So the polls that correctly predicted the outcome were RIGGED and the ones that didn’t were HONEST? > Coworker: _more sadness_ [Geoff Berg](http://blog.chron.com/partisangridlock/2012/11/conservatives-vs-reality/): But the problem with people like Morris and the networks that give him airtime is not just that they’re fools or liars, it’s that the conservative movement is now dominated by people who believe that kind of obvious nonsense without thinking. Just before the election, Twitter was flooded with Fox/Limbaugh/NewsMax/WND types laughing about what a certainty it was that Barack Obama was about retire early. Nate Silver, whose model correctly predicted both the 2008 presidential election and the 2010 midterms? Just a liberal hack in the tank for Obama. Companies like Public Policy Polling which consistently showed Obama ahead in swing states? It sometimes does polling for DailyKos and therefore must be untrustworthy. A Fordham University study released today shows that PPP and DailyKos/SEIU/PPP were the numbers one and two most accurate polling organizations this cycle. Conservative comfort blanket Rasmussen was 24th out of 28. It’s pretty sad the way conservatives, like IIRC OMG!, thought that PPP was _biased_ and Rasmussen was accurate. They could have looked up these groups past accuracy. Instead they used the rule of thumb “On my team/off my team” to estimate their accuracy. Those who use such methods really should be kept away from our nation’s policy levers as well as sharp objects for that matter. ---
**Author:** ![drewtwo99](https://avatars.discourse-cdn.com/v4/letter/d/d07c76/32.png) [@drewtwo99](https://boards.straightdope.com/u/drewtwo99)\ **Post date:** [November 11, 2012, 5:33am UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/119 "2012-11-11T05:33:34Z")
Measure for Measure… you listen to Geoff Berg too??? ---
**Author:** ![Measure\_for\_Measure](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/measure_for_measure/32/557_2.png) [@Measure\_for\_Measure](https://boards.straightdope.com/u/Measure_for_Measure)\ **Post date:** [November 11, 2012, 5:38am UTC](https://boards.straightdope.com/t/nate-silver-538-was-right/640181/120 "2012-11-11T05:38:12Z")
## Er, not really: I just use google news. First I’ve even heard of the guy. :o ETA: You know what I want to see? A time series of Brier scores for the various forecasting outfits. Who cares what they thought on Nov 4? I’d like to know how their forecast accuracy tracked over time. [Previous page](https://boards.straightdope.com/t/nate-silver-538-was-right/640181.md?page=5) [Next page](https://boards.straightdope.com/t/nate-silver-538-was-right/640181.md?page=7)