This prediction, made way back in February (I emailed it to friends at the time which is how I serendipitously came across it now, looking through my sent folder for something else), was startlingly close to dead right. Only Florida was predicted incorrectly, and of all their Romney states, they gave it the greatest chance of going for Obama (38%). And of course it only went to Obama extremely narrowly. (Makes one wonder if choosing Ryan had this one effect, and otherwise all the campaigning was irrelevant.)
Well, I wouldn’t say it makes the entire campaign cycle irrelevant.
At best, it points to the the ineffectiveness of Ryan in influencing the election in any measurable way.
This isn’t just about Nate Silver. Sam Wang at Princeton had a much simpler model that also did extremely well this year. Them and a number of other people did essentially the same thing: Looked at polls, did some amount of statistics, and predicted the election results better than the people who tried to use baseball analogies and ‘momentum’ to see into the future.
The common factor here is that looking at the evidence and taking it seriously gets you the best results. Math is just the tool you use when you are taking evidence seriously. It isn’t about ‘Democratic math’ or ‘Republican math’ or ‘the math you do to make yourself feel better’; it’s about having at least one way to look at the Universe without flinching or pre-judging the results based on emotional bias. Anyone who has that will do better than the people who don’t.
Derleth, I would still argue that the Yahoo Signal prediction is much more impressive than what Wang did. Wang didn’t have any posts on that blog this cycle until June 27th of this year; the Yahoo Signal prediction I linked to was more than twice as far away from Election Day. This is from that June 27 entry on Wang’s blog:
That “Snapshot for 2012” was subsequently unveiled on July 8, and predicted Obama would get 318 EVs and +3.0% in the popular vote. Not bad, but the post is undermined by its bold assertion of a 91% chance Democrats would retake the House (I don’t think Silver ever saw this as likely), not to mention the echoing of the CW that Tommy Thompson was “heavily favored” against Tammy Baldwin in Wisconsin.
Clearly the Yahoo Signal prediction is based on “approach (2)”, as this quote from the link in the OP makes clear:
And if “approach (2)” was up for debate, Yahoo Signal has clearly won this round of that debate IMO. Note that even the fact that they missed slightly low for Obama was covered in their analysis:
A read through the entire post, by the way, will show you that (unlike the way Silver often hedges in both directions) there is no corresponding caveat that “on the other hand, Romney may do better because ______________”. They laid out the results of their model and said Obama may if anything do slightly better than this, period. The next most likely state for him in their probability chart was Florida, and voila: he did win that one narrowly.
Silver’s importance wasn’t in his successful prediction of the election itself but in his successful interpretation of the polls - something the MSM (on both sides of the fence) were unwilling to do for the sake of the 24/7 news cycle narrative.
Their models were based on different things, though. I am wary of the idea of giving them lots of points for making a prediction first.
Silver hedges for good reasons, though: he’s telling you what he expects and why that expectation (even if it’s very likely) could be wrong. Be careful about rewarding false confidence and unsustainably early predictions. Here’s what I do see: Silver was more accurate than Yahoo Signal, and when Silver added Yahoo Signal-type data to his model, it tended to just be noise (Wang did better without that stuff). Does that mean Yahoo Signal is better, or did they just get the right result by dumb luck or for the wrong reason? Those things aren’t always easy to figure out, but I don’t share your confidence here.
In a way, it’s not terribly surprising. Yahoo’s research organization (which is now part of Microsoft Research) is a serious brain trust.
Whoa, how can you say that? Would you consider a weather forecaster who gets the weekend forecast 98 percent right on Monday less accurate than one who gets it 100 percent right on Friday? I wouldn’t, and I’d consider the former a lot more useful when making plans for the weekend.
I don’t think you can just lump it all into a general category of “Yahoo Signal-type data”. It would appear these economists (which Silver is not to my knowledge) did a better job of constructing a model that successfully interpreted that data. There Silver was being a dabbler, whereas they are pros.
I didn’t know that, ultrafilter, and it strikes me as ironic. The hoi polloi that comment at Yahoo are so very brain dead (the comments on the post I linked in my OP are a great example, as even the supportive ones babble on about polls like they are commenting at 538, despite the clear disclaimer that this model is not based on polls). I think this is probably in large part why these guys haven’t gotten more attention from the elite media.
Because it’s true? According to Silver (and Wang), the most likely outcome of the election was 332 electoral votes for Obama. They were both exactly right. Yahoo Signal forecast 303 electoral votes, which those forecasters saw as the next-most-likely result. It’s not like they missed by a mile, but that’s less accurate.
Yes, without question. When I’m getting dressed on Monday, what use is an almost-perfectly accurate prediction of what the weather will be like Saturday? I probably won’t care until Friday, and on Friday I’d just want the best prediction, not the earliest one. The only way that’s useful is if I am packing for a vacation or preparing for a hurricane or something, and in those situations it really doesn’t matter if the forecast is 98% accurate or 100%.
He’s a statistician.
You’re going to have to be more specific about how their model was better. Silver was more accurate (although there’s reason to think his use of economic and other data wasn’t good), and Wang was even more accurate without any of that stuff. Yahoo Signal made an almost-as-accurate prediction much earlier based on different data.
There are basically two questions, I think, in assessing predictions: was it right (all of these models did very well) and was it right for the right reasons? The second one is harder to tease out and I think some of the points the Yahoo Signal writers made - the idea about economic trends being more important than specific figures - are good ones. It should be noted that they were far from the only people making that point. Wang and Silver made a lot more predictions about the Congressional elections at various levels and they both did extremely well, so we have that much more reason to say they were accurate.
I see Yahoo Signal also made a lot of state-by-state vote predictions. How do their forecasts compare to Wang and Silver?
Well, he may have been at the time.
Marley, clearly you and I use weather forecasts rather differently (I almost always go straight to the ten day forecast rather than the “today” or “tomorrow”), but that’s getting off on a tangent. So I’ll just set aside the analogy and say flat out that predicting an election uncannily before one party has a nominee, and without using polls, nine months in advance, is way more impressive than looking at what the polls say will happen the day before.
I also think it is instructive to look more closely at Florida in particular, since that was the only state that “differed” in the respective forecasts. As I said, Yahoo Signal said the model’s forecast at that point was probably a little conservative for Obama if anything, and Florida was the closest Romney state they had to Obama’s side (thus there is a clear inference it would be the next in line, most likely to flip based on that “conservative” caveat). Silver, on Election Eve, had Florida as 50% for Romney, 50% for Obama. (He said on Twitter he had to go to something like the third or fourth decimal point to break the tie, almost surely not a “sig fig”.) I’ll leave it to the reader to determine how much difference there is really between those two forecasts, but one thing is certain: one was made on Feb. 16 and the other on Nov. 5.
By Nov. 5, there were many sources one could look at (as long as one was not distracted by the bleating of blatantly right wing “skewed polls” conspiracy theorists) to see which way the wind was blowing: 538, Wang, HuffPo/Pollster, even the blunt instrument that is the RCP “no tossup states” map. If you count Nate Silver’s tweet, he–unlike them–got Florida “right”, but on his site itself he simply threw up his hands and called it a tie. And it was not necessary to give Obama the win.
But back in the spring, there was only (as far as I know at this point, but am open to learning of more) Yahoo Signal. Regardless of weather analogies, assuming their model keeps holding up I know that in future elections I’m going to want to know what will happen nine months ahead of time, if possible. A political prediction the day before is clearly less valuable than one nine months before. That is so much less time to be “in the know”. By the day before, you can just wait 24 hours and know the results!
I suppose someone could question the value of being “in the know” ahead of time, and in fact some do (they complain that we should just wait to see what happens, as “the only poll that counts is on Election Day”). But first of all, I think there’s an inherent, ineffable value for many of us who follow politics.
And more than that, if these Yahoo Signal guys have “cracked the code”, it means (as a lot of political scientists have said for years) that–at least when incumbent presidents are being challenged by reasonably competent challengers–that the campaign itself probably doesn’t matter except in cases where the factors these guys base their predictions on show a 2000 style Bush-Gore razor’s edge tie. I don’t know about anyone else, but if it’s utterly pointless to follow not just the minutae of a campaign but even the big events like convention speeches and debates, I’d like to know!
And eventual outcome aside, that is an area where there is a clear difference between Silver/Wang/etc. and Yahoo Signal. Silver has gone into depth looking at bounces from debates, conventions, etc.; he had Obama down to only a 56% favourite at one point. Wang says straight out that his aggregation provides a “polling snapshot”. Both of them explicitly or tacitly validate the idea that their forecasts, especially when weeks or months remain until the election, are preliminary and could change drastically based on later poll numbers (implying, then, that campaigning does matter).
But I am really questioning now whether that is true at all. I don’t want to spend countless hours of my time every four years closely following (on Politico, Twitter, etc.) “twists and turns” of a drawn-out process that is in some way a kabuki dance. YMMV.
ETA: On Thompson, I understand that he was widely seen as a favourite–that’s why I noted it as being “CW”, for “conventional wisdom”. But my point was that Wang thus did not in that case cut through the clutter of punditry to give us some insight we couldn’t have heard from any political reporter or columnist. They were wrong, and he was wrong. Also, what about that terrible prediction on the House?
There’s nothing uncanny about this, and the other forecasts were not just the day before the election. I think Wang was in the neighborhood of 332 EVs for Obama in June, for example, and I’m pretty sure Silver said for months before the election that the most likely outcome was 332 electoral votes for Obama.
There were always lots of election predictions. If you managed to forget almost all of them, I think I am jealous.
Again, this is entirely subjective. You are valuing an earlier, less-accurate prediction over a later, more-accurate prediction because, um… you have more time to wonder if the forecast will be more accurate?
You’re getting wildly optimistic here.
The outcome of an election usually looks foreordained after it’s already over. This is also why statements like “this model would have correctly predicted 88% out of the last 500 elections” don’t mean much of anything.
In other words Yahoo Signal is more reliable because they didn’t track the news or update their forecasts? I don’t think that works.
That’s not conventional wisdom, it’s data.
Obviously not, and though you think you sound cleverly snarky you actually just sound incoherent.
The difference in accuracy between the two forecasts is negligible (Silver did not on his blog take a stand as to which way Florida would go). The difference in how far in advance the earlier prediction was made is huge. And your attempt to claim you don’t value lead time for a prediction is most likely simple stubbornness rather than your genuine belief, as you strike me as more sentient than that. Otherwise, going 50/51 on the 2036 election now would not particularly impress you, and you’d also be perfectly happy with 538 if it did not make a prediction until one millisecond before the polls close on Election Day. Come on. Do you get so deeply invested in being contrary that you don’t stop to notice how far down the garden path you’ve wandered in the process? Sheesh.
His model said Obama was going to win Florida very narrowly, giving him 332 electoral votes. He predicted - as Yahoo and some others did - that Florida would be extremely close.
Yes, one was much earlier than the other. Not arguing that point. You are placing a very high value on that earlyness and I am asking you to quantify it, because to some extent you are valuing the early prediction more highly than a later but more accurate one.
I never said I don’t value lead time. I said I don’t value it above accuracy.
I think you should un-exasperate yourself. I’ll ask again: can someone compare Wang and Silver and YS’ forecasts on a state by state level? If Yahoo did better in February, that’d be impressive. If they were less accurate but in the ballpark earlier, again, how much do we value that?
The state-by-state comparison might indeed be interesting. I don’t even have a spreadsheet on my computer (nor do I know how to use one, TBH); so it will not likely be I who will attempt the comparison.
Incorrect. His model’s output is still up at fivethirtyeight.com (or whatever that redirects you to at the NYT), look for yourself. It predicted 313.0 (yes, “point zero”; and no, I don’t know why a decimal is used there) EVs for Obama. In fact, as you will see further down, the model gave the 332 EV scenario only a 20% chance of occurring.
No, this was explained many times: when Silver ran all the simulations under his model, the average result was 313 for Obama. That was not a prediction of the actual result, and that’s why it had a decimal point. There was no realistic scenario where Obama could get exactly 313 EVs.
Which made it - as I said repeatedly - the most likely outcome under Silver’s model. The next most likely outcomes were 303 for Obama (giving Romney Florida) and 347 (giving Obama North Carolina). If you had asked Silver to give just one number instead of a range of probabilities, that number would have been 332 for Obama. That was the right result.
Here is Sam Wang’s final prediction - he actually missed on Florida, too, but was close on a lot of fronts. I don’t think I can pull off a side-by-side-by-side comparison either.
Marley, nothing you have posted in your 2:29 comment is anything new to me. All you are doing is illustrating the vast panoply of caveats and hedges Silver used. Yes, of course 313 is a mean. I have no idea if there is any combination that would give someone that many; but if there is, it is not likely to be one that had any reasonable chance of happening given the probabilities in terms of the “tipping point” state. (Which, btw, Silver also predicted wrongly as he thought it would be Ohio, it then looked like Colorado, and ultimately turned out to be Pennsylvania, despite his only having given it a 3.3% chance of being the one.)
Saying 332 wins a plurality at (barely over) 20% is not the same as predicting Obama would get 332 EVs. In fact, assuming Silver had faith in his model, he would have declined all of the following bets (even though we know he’s a betting man):
SOMEONE WITH MONEY: “Nate, you think Obama’s going to get 332 EVs? I’ll bet you a thousand bucks he will not.”
NATE SILVER: “Not going to take that bet.”
SOMEONE WITH MONEY: “Okay, tell you what. If Obama gets 332 EVs, I’ll give you $2,000; if he doesn’t, you give me $1,000.”
NATE SILVER: “Sorry, can’t do it.”
SOMEONE WITH MONEY: “Wow, you drive a hard bargain. Okay, I’ll put up $3,000 that you get if Obama gets 332 EVs; you still only have to pay me $1,000 if he doesn’t.”
NATE SILVER: “Nope, still won’t take your bet.”
That may qualify in some people’s books as “predicting 332 electoral votes”, but not in mine.
ETA: More simply, if Silver had wanted the number 332 to be a prediction, to be a yardstick he’d ultimately be judged on, that number would actually appear somewhere in the sidebar to his blog. It does not.
Give Wang credit: he did, unlike Silver, put up a prediction that included a number of EVs that could actually happen. His was very close…and, of course, the same number Yahoo Signal put up nine months earlier.