Is Nate Silver just in an "I can never be wrong" position.

[jtgain]

Nate Silver today has Obama at a 3 to 1 chance of winning reelection. He is far outside of polling organizations that show many states in play and national commentators talking about how razor thin this margin is.

But, he's "just presenting statistics," right? If Romney wins, can't he just say that he told all of us that there was a 25% chance of that happening, and it happened! How could someone like Silver ever be proven a fraud* or just plain wrong?

*I'm not suggesting he's a fraud. He seems to have his shit together, but let's pretend that he was an old fashioned snake oil salesman. What would prove that?

ETA: Should be question mark in the thread title...can't edit it..

[iiandyiiii]

Originally Posted by jtgain

Nate Silver today has Obama at a 3 to 1 chance of winning reelection. He is far outside of polling organizations that show many states in play and national commentators talking about how razor thin this margin is.

He's not far outside the polling- the national polls, taken altogether, show a near-tie, but the state polls (again, taken altogether) show Obama with a stable and significant advantage in enough states to get ~290 EVs (and very close in a few other states).

But, he's "just presenting statistics," right? If Romney wins, can't he just say that he told all of us that there was a 25% chance of that happening, and it happened! How could someone like Silver ever be proven a fraud* or just plain wrong?

I don't know how he could be proven a fraud, but his credibility would certainly take a hit if Romney wins (assuming the percentages stay about the same until election day). But he's not brand new at this- he's taken seriously, and his predictions are taken seriously, largely because his predictions in 2008 and 2010 were very close to the actual election results.

*I'm not suggesting he's a fraud. He seems to have his shit together, but let's pretend that he was an old fashioned snake oil salesman. What would prove that?

I don't know. If he's a fraud, then he's a fraud who somehow managed to predict very accurately the results for the elections in 2008 and 2010.

[Gorsnak]

Run the election 100 times and see if the distribution matches.

[jtgain]

Originally Posted by iiandyiiii

I don't know how he could be proven a fraud, but his credibility would certainly take a hit if Romney wins (assuming the percentages stay about the same until election day).

Why is that the case? Flip a coin twice and have it land on heads both times. Does my credibility take a hit if I tell you there is only a 25% chance of that happening?

[Happy Lendervedder]

I'm not suggesting he's a fraud. He seems to have his shit together, but let's pretend that he was an old fashioned snake oil salesman. What would prove that?

The free market. If his product (election predictions) suck, no one's gonna buy them.

If he's "wrong," I imagine he'll go back, look at where he messed up, re-calibrate his methodology, and try again. If he is exposed as a fraud, snake-oil salesman, or just plain ineffective at predicting election outcomes, his gravy train will ride away. It's in his best interest to get it right.

And if he does get it "wrong," it's not in his best interest to just throw his hands up and say "Welp, I said there was a 25% chance Romney would win." I honestly think he would own up to the miscalculation, figure out what really happened for his prediction to be so off, and hopefully move forward. He's not this highly regarded because he's been largely wrong. And where he has been wrong, he looks at the variables that caused his predictions to be off so he can correct his methodology.

[Ludovic]

Originally Posted by Gorsnak

Run the election 100 times and see if the distribution matches.

Indeed, if he stays in this business for 30 years and successfully predicts the winner of each presidential race by declaring them the 75% favorite, then he isn't all that he's cracked up to be. He should have "lost" some and had been estimating the percentage too low.

But we don't have enough data at this point to know, if we're just looking at Presidential races.

[Jophiel]

Originally Posted by jtgain

national commentators talking about how razor thin this margin is.

Just to make an initial note, the national polling can be "Razor thin" but you're still going to have one winner and one loser (disregarding a 269/269 split). In the states that matter most for getting 270, Obama has had slim but consistent leads. If the average is constantly putting him +2 in Ohio, it's very likely that he is, in fact, +2 in Ohio and while +2 might be a slim lead, it's a consistent and reasonable one. More so than assuming every poll must be off by three and it's really Romney +1.

No single election will prove Silver wrong. Let's just say that on election day, the model shows Obama getting exactly 270 EVs. And let's say Obama loses Ohio by a handful of votes but gets every other state the model predicted. Does Obama losing the presidency show Silver's model is garbage or Silver is a fraud? I'd say no... you'd need a string of failures (like wrongly predicting several states by notable margins) to start claiming the model is poor.

The other obvious issue is "Garbage In, Garbage Out". If Romney wins Ohio by 7 points, there was no hint that that would happen and it would be unfair to blame it on Silver's model per se. That would be a more systemic issue with polling in general. You could say Silver's model is no good because you have no accurate sources of data but that's different from an innate flaw with the model itself.

[LinusK]

That's the great thing about manufacturing odds for one-time events - you can never be proven wrong.

Still, what Andy said was right. He isn't outside the mainstream. Everyone who follows the polls agrees the polls from the swing states give Obama an advantage, even though the national polls are showing they're roughly tied.

If CNN or whoever is saying something different it's because it's easier, or they don't want to piss anybody off.

[iiandyiiii]

Originally Posted by jtgain

Why is that the case? Flip a coin twice and have it land on heads both times. Does my credibility take a hit if I tell you there is only a 25% chance of that happening?

Because credibility is about perception, and people will perceive that he was wrong.

[jtgain]

Originally Posted by Happy Lendervedder

And if he does get it "wrong," it's not in his best interest to just throw his hands up and say "Welp, I said there was a 25% chance Romney would win." I honestly think he would own up to the miscalculation, figure out what really happened for his prediction to be so off, and hopefully move forward. He's not this highly regarded because he's been largely wrong. And where he has been wrong, he looks at the variables that caused his predictions to be off so he can correct his methodology.

I guess that's my point. He wouldn't be "wrong." He said that there was a 25% chance that Romney would win, and he did. There is no "wrong" there anymore than if I told you before flipping two coins that there was a 75% chance you wouldn't get two heads in a row.

Why should he then have to recalculate or do anything else?

[jtgain]

Originally Posted by iiandyiiii

Because credibility is about perception, and people will perceive that he was wrong.

But if he is the statistical guru of politics, the perception would then be wrong.

[Brainiac4]

The thing is, he's not making just one prediction. He's making a whole bunch of them - forecasting outcomes down to the congressional district level - and so you have more than one prediction to judge him on. Even if Obama wins, Nate could still be "wrong" if he gets the electoral vote totals wrong - he's currently predicting Obama with 294 EVs (as of late Monday), and if that's his prediction going into the election and Obama gets 315, Nate would have missed.

[Exapno Mapcase]

It's not about getting it right. I've been saying for a year that Obama will win a close election because there are too many states that he won last time for Romney to flip.

Who cares that I said that? It's a gut feeling, based on nothing more than my living through a lot of elections.

Silver, however, has published a methodology that states line by line what he considers important, what specific numbers he evaluates, and how he weights and treats them. It's that public methodology that is being tested. It's like making your source code public. Everybody who is qualified can test it for themselves. If it doesn't work, everybody knows why.

There are others who make wildly different predictions based on different assumptions about different numbers, most notably a pair of University of Colorado professors whose software has Romney winning with 330 electoral votes.

They both can't right. Not even "right." One will be wrong, and I mean wrong. The actual results may wind up somewhere in the middle, of course, but the odds are that one of them will be totally discredited. It's simply not true that either is in a real "I can never be wrong" position. That misunderstands the situation.

[GIGObuster]

One thing that many on the right forget is that Nate Silver is also respected because he demonstrated before how and why some pollsters in the past were indeed selling snake oil, one of the pollsters busted by Silver was a pollster that DailyKos used to look at, (they dropped that pollster after the Silver expose) so one of the reasons he is well respected is that he also has done a service by continuing to clean up the wells from all sides.

(Back on July 2010: )
http://www.esquire.com/blogs/politic...ccuracy-070110

The stats-geek world is having its equivalent of a political sex scandal this week: Daily Kos founder Markos Moulitsas accused his pollster, Research 2000, of fabricating its numbers. Moulitsas is suing over the embarrassment; meanwhile Research 2000 is trying to silence Nate Silver, who questioned the organization's credibility. Because we did not major in mathematics — and because the new reality of America's polling universe are changing the way America trusts information — we spoke with Dr. Brian Scaffner, a political science professor at the University of Massachusetts, Amherst, and contributor to trusted site Pollster.com.

It's surprising to see this happen again. Last year Nate Silver produced all these statistical analyses proving that Strategic Vision made up their data; further research into them proved their office address was fake as well. There has been an explosion of these second-tier survey organizations, and not all of them can possibly support themselves. They simply can't afford to produce what they say they produce.

[enalzi]

Originally Posted by Brainiac4

The thing is, he's not making just one prediction. He's making a whole bunch of them - forecasting outcomes down to the congressional district level - and so you have more than one prediction to judge him on. Even if Obama wins, Nate could still be "wrong" if he gets the electoral vote totals wrong - he's currently predicting Obama with 294 EVs (as of late Monday), and if that's his prediction going into the election and Obama gets 315, Nate would have missed.

That's actually just the average EVs of all his simulations. His highest probability is 330 evs, at 14%.

[Hershele Ostropoler]

Originally Posted by Jophiel

No single election will prove Silver wrong.

What Braniac4 said, but also this. It's not the case that Silver can never be wrong, but he'd have to badly screw up a couple of elections, or slightly screw up a shitload of elections, in order to be wrong. But that's not just Silver, that's anyone who makes predictions that they claim to have a probability other than 100%.

Silver is getting a lot of attention on that score, I think, because he works for the NY Times and he's predicting an Obama victory and it fits into the narrative of "the liberal media tries to have its own facts."

[El Zagna]

A candidate defying the odds would not be troubling to Silver. If NO candidates defy the odds, well, that would be a problem.

On his website he gives the odds for a lot of different scenarios. I am sure that after the election he will crunch those numbers to see if his percentages match up with reality and adjust his model accordingly.

If he is "wrong" about Obama, I'm sure a lot of people will abandon him, and to them I say good riddance. As long as the win/lose ration is consistent with his model, I will still follow him closely.

[Euphonious Polemic]

Originally Posted by jtgain

He is far outside of polling organizations that show many states in play and national commentators talking about how razor thin this margin is.

He's really not that far outside. And he takes all of these polling organizations into account in his model. By taking in ALL the data, the theory is that he can then be more accurate than a single polling organization can hope to be.

And national commentators talking about how "razor thin" the margin is? I think for the most part they are trying to attract people to view their own particular news station. It's like getting viewers to watch your football coverage - you don't get commentators the week before talking about how it's going to be a blow-out and a waste of time to watch. They talk about what a great, close game it will be and how everyone better tune in to watch because you never know who might win.

Most news networks treat the election like a sporting event. Many people do as well, more's the pity.

[Leaper]

Besides which, he's not running models on just the Presidential election. He's doing it for every major federal election there is. So that's a TON of opportunities to prove or disprove the worth of his system. That's why his past record in '08 (and, IIRC, '10) is so remarkable.

[RickJay]

Originally Posted by jtgain

Nate Silver today has Obama at a 3 to 1 chance of winning reelection. He is far outside of polling organizations that show many states in play and national commentators talking about how razor thin this margin is.

3 to 1 is a razor thin margin.

I know it SOUNDS great, but for Romney to have a 1 in 4 shot at winning, the polls have to be very, very close. 3-to-1 is not by any stretch of the imagination a commanding lead.

What do you think Michael Dukakis's chances at this point in 1988 were? Probably one in a hundred. Mondale in 1984? Dole in 1996? Not even that; they were absolute toast and had no realistic shot at all.

[astro]

Nate Silver is a limp wristed queer. You can ignore his faggoty polls.

[Lantern]

Ultimately the 538 model depends on the polls. If the polls are wrong the model will be wrong. The model does have some bells and whistles but they aren't enough to overcome a systematic problem in the polls. 538's final state-by-state prediction is probably going to be very close to that arrived by taking a simple average of the last 10 polls in every state.

A sophisticated modeling approach is more useful in races where polling is fairly light like primaries. IIRC in the 2008 Democratic primaries, Nate significantly improved his forecasts by combined polling data with state demographic data. A model also helps in understanding a race which is some months away. I believe every single state in the 538 model today is leaning the same direction as June which is impressive considering that there was much less polling data then and the model relied more heavily on economic variables. To understand exactly how state-by-state unemployment numbers will affect an election you need a good model.

[iamthewalrus(:3=]

Originally Posted by Brainiac4

The thing is, he's not making just one prediction. He's making a whole bunch of them - forecasting outcomes down to the congressional district level - and so you have more than one prediction to judge him on.

This. Silver is making lots of predictions. You can't judge his accuracy based on just one of them. But you can get a good idea of whether his predictions are statistically significant themselves by looking at the lot.

This is why he's functionally different from those meaningless oracles, like the high school football team that predicts the presidential election since whenever, or the octopus who picks a March Madness bracket.

[Gangster Octopus]

Also, keep in mind, a small advantage in the polling popular vote for a state does not necessarily equal a small percentage chance of winning. For example, if Obama is polling a state at 52%, with a 2% margin of error, means Obama should win the start 66% of the time.

[njtt]

Originally Posted by jtgain

But, he's "just presenting statistics," right? If Romney wins, can't he just say that he told all of us that there was a 25% chance of that happening, and it happened! How could someone like Silver ever be proven a fraud* or just plain wrong?

How is he different, in this, from any other pollster? Why are you singling him out?

[Simplicio]

Originally Posted by njtt

How is he different, in this, from any other pollster? Why are you singling him out?

"Other pollster"? Silver isn't a pollster.

[Measure for Measure]

Nate Silver isn't the only person modeling the elections. Sam Wang of Princeton Electoral Consortium is another: he tends to give much higher odds for an Obama victory. After the election, the various polling aggregators (for that is what they are) can discuss where they went wrong and right. Silver has 50 probabilities for 50 states over 2 elections: that's a sample of 100, enough to do some crude probabilistic matching perhaps.

The time series of the reported probabilities can be compared across analysts: it seems to me that Sam Wang will fall down under this criteria.

Meanwhile, when faced with all this mathematical rigor, conservatives are predictably butthurt. Never mind that Silver hasn't touched his model since last Spring and no substantive objections have been offered. It must be librull bias!

[Little Nemo]

Originally Posted by Exapno Mapcase

There are others who make wildly different predictions based on different assumptions about different numbers, most notably a pair of University of Colorado professors whose software has Romney winning with 330 electoral votes.

I think the Bickers and Berry prediction is weak. They based their prediction on economic indicators not poll results. They tracked indicators like unemployment and income and then correlated those with the outcomes of past elections.

Now I'm not saying that's valueless but I think that approach has its limits. People do make voting choices for economic reasons but there are also dozens of other reasons. And more critically, I think it's wrong to make a prediction based on what you think people should believe and ignore contradicting data on what people say they believe.

[Jas09]

I don't think you need to be concerned about Silver being treated too lightly if his predictions prove incorrect. There is an entire media machine ready to tear him apart. If he predicts even a 51% Obama re-elect chance and Romney actually wins Silver will get way more than his share of mocking and derision, and will likely go into the next cycle with very little national credibility.

Predictions made by math and science are treated with far less mercy if they are wrong than predictions made by intuition, woo, or "gut feelings". Just look at the jailed earthquake scientists for an example of this. Or, conversely, Dick Morris has been wrong more times about his political gut feelings than anybody in the world, but still has a job and his predictions are still given credence.

[Hershele Ostropoler]

Originally Posted by Jas09

Predictions made by math and science are treated with far less mercy if they are wrong than predictions made by intuition, woo, or "gut feelings".

I don't know if that's the situation here. Nate Silver is perceived as liberal, and he's predicting success (in aggregate) for Democrats, who are also perceived as liberal, and so people, particularly conservatives and political hipsters, are skeptical that he's even using a model.

[Marley23]

Originally Posted by Hershele Ostropoler

people, particularly conservatives and political hipsters, are skeptical that he's even using a model.

Conservatives are skeptical because he's saying something they don't want to hear (and of course liberals like the fact that he's saying something they approve of), and I think a lot of political reporters resent the fact that he's making predictions based just about entirely on polls and hard data without a lot of the other speculative bullshit that has come to pass for political reporting. I can't vouch for Silver's statistical methods, but they're a tool that can be used to understand what's going on. If it's this or watching a debate and making guesses about what viewers will think about the body language of the candidates, I know what I consider more informative.

I don't think the criticism in the OP is reasonable. It's like criticizing a weatherman if he tells you there's an 80 percent chance of rain and it doesn't rain. Maybe the station's weather satellite stinks, or maybe he was right - because even if there's an 80 percent chance of rain there's a decent chance it's not going to rain, and it doesn't make sense to demand certainty when we can't know for sure what's going to happen.

[DigitalC]

Even if Obama won by taking Florida and losing Ohio we would start to question Silver's model.

[Jas09]

I think one thing that would make me a bit skeptical is if "simpler" models (like a straight average of the polls) performs better. That would at least imply that his weighting and adjusting is not adding value to the predictions.

[bup]

Originally Posted by DigitalC

Even if Obama won by taking Florida and losing Ohio we would start to question Silver's model.

That would be a more valid criticism than whether or not he 'calls' the presidency correctly. At least that's two events he predicted to be unlikely in the opposite directions. (Today he has Ohio at 73% Obama, Florida 64% Romney).

[MsWhatsit]

Originally Posted by Marley23

I think a lot of political reporters resent the fact that he's making predictions based just about entirely on polls and hard data without a lot of the other speculative bullshit that has come to pass for political reporting.

God. Yes. Seriously, I could not agree with this more.

Look, even if you think that Silver is a partisan hack, he's completely transparent with his methodology, so you can easily go and see exactly what he's doing and say, "OK, I think you are inappropriately weighting this set of polls" or whatever. It's not like he's just making up a bunch of bullshit.

[Enginerd]

Originally Posted by Hershele Ostropoler

I don't know if that's the situation here. Nate Silver is perceived as liberal, and he's predicting success (in aggregate) for Democrats, who are also perceived as liberal, and so people, particularly conservatives and political hipsters, are skeptical that he's even using a model.

I don't understand this criticism. Silver also predicted 34/36 2010 Senate races correctly (24 Pubs and 10 Dems), and the two he got wrong were states that elected Dems where his model said Pubs (Colorado and Nevada). Do the people criticizing him now think he was just trying to set everyone up two years ago?

Well, ok, I do understand the criticism. It's just wrong.

[Marley23]

Originally Posted by Enginerd

I don't understand this criticism.

You can't understand it with your brain. You feel it with your gut.

[Munch]

Originally Posted by Measure for Measure

Never mind that Silver hasn't touched his model since last Spring and no substantive objections have been offered. It must be librull bias!

Do you have a hard link for this? I'm also pretty sick of the constant stream of bullshit about Silver "massaging the numbers in favor of his preferred candidate" and would like something to point people to who are unwilling to investigate his methodology.

[Sitnam]

Originally Posted by jtgain

*I'm not suggesting he's a fraud. He seems to have his shit together, but let's pretend that he was an old fashioned snake oil salesman. What would prove that?

Being off the mark consistently in the past. He has a reputation that's well deserved for accuracy, but I guess it's possible he could throw this prediction on purpose....for what reason I have no idea.

[CJJ*]

For me, the least interesting thing from Silver is his odds of Obama winning the election; I find his exploration of the data far more intriguing and thought-provoking. Focusing on the odds is like focusing on how many stars Roger Ebert gives a movie and ignoring his excellent film analysis.

[LinusK]

Originally Posted by DigitalC

Even if Obama won by taking Florida and losing Ohio we would start to question Silver's model.

No, you'd question the polls - that show that Obama's ahead in Ohio and behind in Florida.

[Gangster Octopus]

BTW, no one will be more critical of Silver if he turns out wrong than Silver himself, in the sense that he will actually attempt to find out where the error was. I am currently reading his book The Signal and the Noise and one of his themes is that folks need to accept that there is uncertainty in prediction.

[Omg a Black Conservative]

The only thing worse than Nate Silver are his (cult like) followers, who disregard certain points because "Nate says otherwise".

[Airbeck]

Originally Posted by Omg a Black Conservative

The only thing worse than Nate Silver are his (cult like) followers, who disregard certain points because "Nate says otherwise".

The only thing worse? Can't think of a single thing on earth worse than Nate Silver or his followers, huh? Wow. Partisan butt hurt much?

Also, it isn't that Nate says otherwise, its that the numbers say otherwise. Nate is a stats geek. All he cares about are the numbers. You aren't mad at Nate, or at those that follow his blog, you're just mad at the numbers. You can admit it. Its ok.

[iiandyiiii]

Originally Posted by Omg a Black Conservative

The only thing worse than Nate Silver are his (cult like) followers, who disregard certain points because "Nate says otherwise".

As opposed to disregard the state polls, which have a record of accuracy (as a whole), just because it "doesn't fit" or whatever?

[Marley23]

Originally Posted by Omg a Black Conservative

The only thing worse than Nate Silver are his (cult like) followers, who disregard certain points because "Nate says otherwise".

This is a threadshit.

Originally Posted by Airbeck

The only thing worse? Can't think of a single thing on earth worse than Nate Silver or his followers, huh? Wow. Partisan butt hurt much?

And this is a personal insult. Both of you are encouraged to take this to the Pit because it's not appropriate for this forum.

[Uncle Jocko]

Originally Posted by Marley23

Conservatives are skeptical because he's saying something they don't want to hear (and of course liberals like the fact that he's saying something they approve of), and I think a lot of political reporters resent the fact that he's making predictions based just about entirely on polls and hard data without a lot of the other speculative bullshit that has come to pass for political reporting.

So, you're saying he's blinding them? With SCIENCE?

"She's tidied up, and I can't find anything!"

[Airbeck]

I apologize for the butt hurt remark and retract. Although I believe I've seen that phrase used several times recently in this forum without any mod comments. Nevertheless, consider it withdrawn with apologies.

[MOIDALIZE]

Not that I take Nate's detractors seriously, since they're obviously engaged in a smear campaign (they didn't seem to have any complaints in 2010), but he is making state-by-state predictions. That's 50 different results we can evaluate. So if he has Obama winning Ohio by a margin of 2.1%, and Romney wins by a margin of 2%, that's a big enough swing to call his model into question.

If the wingers really want to get angry, they should check out Sam Wang's prediction.

[Omg a Black Conservative]

Originally Posted by Airbeck

The only thing worse? Can't think of a single thing on earth worse than Nate Silver or his followers, huh? Wow. Partisan butt hurt much?

I'm guessing no. But what do I know, right?

Also, it isn't that Nate says otherwise, its that the numbers say otherwise. Nate is a stats geek. All he cares about are the numbers. You aren't mad at Nate, or at those that follow his blog, you're just mad at the numbers. You can admit it. Its ok.

What numbers? People constantly say this, probably because they seemingly believe the people who criticize Nate's "methodology" don't understand polling nor statistics, but Nate looks at the top line of any given poll, applies an arbitrary weight to it and then uses some super secret formula which no one knows to come up with his final percentages. Hell, I could do that, too. Silver's popularity seems to come from his accuracy in the 2008 election (an election, mind you, where he received internal polling from the Obama administration). Yet the same people who adore Silver seem to ignore the fact that he was just okay in 2010. In fact, he was no more accurate than Rasmussen (who everyone around here loves to denigrate).

Now never mind the fact that Silver doesn't really delve past a polls top line, even looking at an aggregate of all the polls, Obama is about at 47%/48%, significantly lower than where he is on Silver's model. When do we break out the "Nate Silver vs. the World" headline?