I saw that Rasmussen move, and Fox News .com had an opinon piece on that and how “they” are in trouble if people dont get out and work
A slight uptick this evening. Silver now has Obama at 81.1%
And again today, up to 83.7%.
This just in tomorrow…Silver has Obama at 203%
I laughed.
That was by Dick Morris, who is about as insane a Republican partisan as walks the earth. Mitt Romney isn’t as partisan as Dick Morris.
Of course, the article is presented in such a fashion as to state that it is merely the opinion of Morris and not of Fox News… but that print is really small.
Nate Silver’s word, the 538 blog, must be our ultimate authority. Nate Silver says it, I believe it, and that settles it.
I am going to admit that if this happened, I would assume that I was wrong about how to do math.
Here’s what Nate Silver considers a pretty solid conservative criticism of his predictions:
http://www.pointoflaw.com/archives/2012/11/my-election-prediction.php
I like 538 and over the past few weeks have read it nearly daily. I do think the OP raises an important question, though, which could be rephrased as: what type of metric could we use as a benchmark to confirm or deny the validity of his methods? Obviously just the up-down, binary question of whether Obama does or does not win is not enough, especially given the hedge (even now, 17 percent is quite a bit–about the same as the chance in poker that you’ll hit your flush on the river if you have a flush draw on the turn).
It is often pointed out that Silver predicted which way 49 out of 50 states would go in 2008. That always sounded impressive enough for me, but one of his critics noted that the RealClearPolitics (RCP) list of a plain old average of polls for each state also predicted those same 49 states. So, hmmm.
I notice that his model currently predicts that Obama will receive 50.6 percent of the popular vote nationwide, with Romney expected to get 48.3 percent. These were the same exact figures the model predicted when it began for the season on May 31. The RCP polling average on that date, however, was quite different–so clearly there is far more to his model than some linear extrapolation from poll averages (he has written that there are economic and other factors built in, which are gradually made less and less important relative to the polls as Election Day becomes nearer). Furthermore, if you ignore the “noisy” periods right around the conventions and the debates, the model has consistently stayed within a half point (and usually much closer than that) of those numbers, even while the RCP poll average fluctuated much more widely.
So I would venture to say that if Obama’s final tally in the national popular vote is between 50 and 51 percent, and even moreso if it is smack in the middle of those numbers, then Silver comes out looking very good (although there’s always the chance he got lucky, of course). If Obama’s share is 49.5-49.9 or 51.0-51.4 that’s still respectable. If Obama gets a greater share of the vote, the model will not in the end look all that impressive, but it will at least have predicted the right winner. If Romney barely edges Obama out, I’m going to say that’s fairly shaky but not a total disaster for Silver (for the country, maybe). If however Obama nets less than 49 percent of the popular vote it’s a clear fail for Silver’s credibility as I see it.
And with good reason. The possibility that state polls are systemically statistically biased cannot be dismissed out of hand and there are real reasons to suspect they might be. The systemic source of error I can imagine is that the means of correcting polls so they reflect local demographics is, by necessity, different than the means used for the national polls. Such a difference could amplify any subpopulation response variability. Is Silver’s quantification of the possibility of state bias towards Obama on any real basis or is it arbitrary? Is it higher or lower? It really is hard to know given that the nature of the demographics, the character of each of the subpopulations, have each varied over time and varies state to state.
To state that the biases are equally likely to be occurring in either direction to me seems a bit glib. To me it seems a bit more rational to assume any bias favors the side consistently ahead compared to another method: the possibility that the state polls are biased to Romney then implies that the Gallup Rasmussen polls are outrageously biased to Romney by historically unprecedented amounts, something fairly improbable; the possibility that they are biased to Obama merely implies that Gallup/Rasmussen are accurate, which is not unbelievable and statistically less unlikely.
Yes it helps that other national data is not so far off from the state aggregations, but still.
One could take it one level further: how do his popular vote predictions match up state by state? And of course include the Senate race predictions. Any state that he has “robust” datasets in should result in pretty good confidence intervals. We can judge that against other aggregator methods and against individual pollster popular vote predictions.
End of the day however each election will be a different one time event. Judging him is ultimately more akin to judging a stock picker/fund manager than it is evaluating a scientific hypothesis. Outperforming the index is good but it does not guarantee future results as the markets change.
If I were to evaluate Silver’s accuracy post-election, here’s how I would look at it: He offers a state-by-state analysis of the projected vote share; e.g. right now he has Florida with Obama at 49.6% of the vote, Romney at 49.8%, +/- 2.9% error; Iowa he has at 51.0% Obama, 47.7% Romney, +/-3.5% error. I would look at the actual percentages the candidate got state-by-state and see if they fall into the prediction interval suggested by this, instead of whether he got the point estimate right. I’d count against him anything that did not fall into his suggested range.
Obviously this could lead to cases where either a Romney or Obama win in that state might count as a “good prediction” (e.g. where in FL you have basically a statistical tie) if the results still fell within the range. But I think you can’t hold that against him if the polling data tells you it’s essentially a coin flip and the final result is a close one. OTOH, if he’s predicting a result like Pennsylvania as an Obama victory (52.2% Obama, 46.7% Romney +/- 2.8%, i.e. 97% chance of victory for Obama) and say for example Romney ends up with 51%–outside of the prediction interval–then you count that as a failing of the model. Perhaps due to state polling bias or whatever, but the point being if the model didn’t correctly account for that, that would be a ‘miss’ for him.
I’d be interested in seeing how he does as a scorecard using an approach like this.
I think the most important line from the PointOfLaw critique is this one:
It’s possible that Mr. Silver is getting it wrong. But it’s highly unlikely that he’s not working very hard to get it right.
What would help Silver’s credibility is if he stopped making narratives. Because it makes it less obvious what differentiates him from the pundits. Of course, narratives are how humans communicate ideas, but at the very least he should be much more obvious that his primary product is the numbers. I think he’s been doing that recently, anyway.
Assuming a 95% confidence interval, two of the states should fall outside of the range.
I’m not sure what you’re referring to. His primary product is his blog, whose mission is to " help New York Times readers cut through the clutter of this data-rich world." The blog is devoted to rigorous analysis of politics, polling, public affairs, sports, science and culture, largely through statistical means. In addition, FiveThirtyEight provides forecasts of upcoming presidential, Congressional and gubernatorial elections through the use of its proprietary prediction models.
FiveThirtyEight was founded by Nate Silver in March 2008, and was licensed by The Times in August 2010. It is produced in conjunction with The Times’s graphic and interactive journalists and its team of political editors, correspondents and polling experts. http://fivethirtyeight.blogs.nytimes.com/about-fivethirtyeight/
Narratives are fine, provided they are grounded on data, as opposed to entertainment value. But maybe I misunderstand what you mean by “Narrative”.
Furthermore, Nate says that his model is over-rated: in other words, there are a number of statistical models out there and I personally find it unlikely that his will fit the electoral outcome best. (As it happens, his tends to lean towards Romney.) But he’s a lucid writer and explains statistics and probabilistic thinking to the laymen rather well. Pushing an empirical approach to current events provides a real service for those wanting to be informed, as opposed to reassured or entertained.
ETA: The conservative law blog gives Obama 60% odds, while Nate gave the President 80% at the time of the post (85% now, 2 days before election). Frankly, that seems to me to be the basis for a reasonable, data-driven, difference in opinion. Go reality!
The Indianapolis Colts took a 3 point lead in the 4th quarter of their game against the Miami Dolphins, and held on for the win.
Obviously this is because CBS was in the bag for them.
The real story is that the Redskins lost their last home game before the election,which means a Republican victory. It’s science.
I believe you’re right–the multiple comparisons problem, and admittedly I’d overlooked that. So what’s the fix–something like a Bonferroni correction? Or do you say you allow for a small number of false positive results due to random chance given 95% confidence, but if he has too many such “misses” then say there’s a problem with the model?
Except the last time there was an incumbent president up for re-election, they also lost, so, perhaps the trend has reversed course now. (thought that’s the only time the Redskin trend hasn’t been followed).