GOP would pick up 7 seats according to the polling average if the election were held today. Wang is not actually going by polls. For example, in Alaska he has Begich ahead. Begich is behind by 6 in the first reliable poll taken since Sullivan won the nomination. He’s also counting Kansas as in the bag for Orman, despite the fact Taylor will still be on the ballot and despite the fact that are no polls since that decision showing the state of the race. Finally, he has Hagan and Pryor ahead in their races. They are not ahead, they are behind:
No one, and that includes Nate Silver, has a better track record than Wang. The method for these senate races:
The reason for the divergence between his approach and others (including 538) is that he does not adjust the data based on some “fundamental analysis”; he let’s the polling data fall as it falls.
and those other indicators, “the fundamentals”, are the maxims that the pundits like to discuss - midterms tend not to favor the sitting President’s party; it’s reasonably safe to assume that turn out will favor the GOP … hence 538 adjusts the polling data results to reflect those “other indicators”, the “fundamentals.”
Wang may use hallucinogens. I don’t know his recreational habits. But he worships at the altar of the numbers and what he says the polls say, the polls say, whatever he does in his free time. 538’s take is that the poll meta-analyses are more reliable with some (mostly) fundamentals based tweaking. But Silver’s emphasis is on how uncertain things are; his edge to the GOP is very slight and his hedges are large. IMHO the line between tweaking the polls based on “other indicators” and punditry gets fuzzy.
Well, let’s keep in mind that Wang is the outlier this time. I was specifically disagreeing with Silver in 2012. Now you are.
Wang’s track record is the best. The problem is that his model relies on polling that has not been available to the same extent as in past years. As Silver has pointed out. If Wang’s model is dependent on good polling and there isn’t much of it, and Wang doesn’t notice that, then there’s the weakness. Plus, as I said, he seems to be ignoring polling. There really is no excuse for him to have Pryor ahead, and his prediction on Alaska should change within a day or two.
The reason to use “special sauce” is that the farther out you are, the less reliable the polls are. For example, an incumbent who does not yet have an opponent is going to poll better than an incumbent who does. That’s why Democratic incumbents all appeared to be winning two months ago.
Once you get into October, of course, Wang’s model is probably superior, but Wang doesn’t have the number and quality of polls right now. And there’s probably no penalty for being wrong in September, because by November Wang’s polls will show him pretty accurately where the race is. But I’ll still remind you all why Wang was wrong in September, and it’s because his model doesn’t get superior until we get really close to the election. WAng will say that “things changed”, but Silver and the other prognosticators already predicted the change.
I hope Wang’s right, but I’m certainly not as confident that he is. It’s still probably a bit too early for this to be a particularly good prediction either way, and presidential elections are inherently easier to predict than control of congress/senate – the polling is better and more frequent, and there are fewer variables (i.e. contests). We’ll see what they say into October.
Honestly I am not going to second guess his analysis of the polls (albeit the Alaska poll just out will adjust his “if held today” number some, and the prediction of November a smaller bit). I do think you misread his takes. He does not count “Kansas as in the bag for Orman” for the purposes of the big prediction. His exact quote before the recalculation that moved the number to 70% was
Hagan and Pryor are both about as close to coin flips as you can get and he handles them as such.
To illustrate why though why the RCP moving average gives the 0.8% lead to Tillis while his approach gives a 2% lead to Hagan recognize that he drops after 3 polls out. So the +1 most recent Tillis replaced a +5 Tillis in his moving average while RCP still counts both.
I also don’t think anyone is really much of an outlier right now. 64% odds one way or 70% odds the other are all pretty much saying it is going to be down to the wire and non-random walk events that might occur nationally could change things mightily across the whole stage. A small nation-wide shift either way changes everything. No one has a crystal ball about that. And more polling data as things get closer might change predictions too. Silver is saying it is very very uncertain. I really cannot disagree with that.
Foreign events are also making things very uncertain right now. If the President has to go to war, there will be a rallying around the Commander-in-Chief effect.
I wasn’t referring to Republicans, I was referring to voters. But I’m not too worried, because the President doesn’t decide anything in less than six months.
Yep. That’s what Congress is supposed to do. The media too. I realize the current President is a creampuff, but he shouldn’t have sought the job if he didn’t want the oversight.
You rally around the COmmander-in-Chief when he does things. When he just lets things happen to him, not so much. I backed the President in Libya, I back him in Iraq, and i’ll back him in Syria, Iran, Russia, wherever he decides to go. Benghazi was a clear failure, and as long as there are unanswered questions it should be investigated. Pearl Harbor was investigated during the war, no reason not to investigate Benghazi.
Yes, but he’s not on hallucinations. And figuring out whose modelling approach is best will require serious investigation: I’m not even sure it’s statistically possible as of now.
Two things though.
Wang was saying back in late May that polls outperform fundamentals. That strikes me as almost an extraordinary claim (not quite though). It certainly conflicts with my intuition. So I’m discounting his model further, notwithstanding its strong 2012 performance. If the models show collapsing support for the GOP in another month though, that will be a feather in Sam’s cap.
Wang, again from you May article: “But in my experience, any contest with probabilities between 20 percent and 80 percent should be regarded as a toss-up, with no solid favorite.” Spend some time playing monopoly and you are likely to agree: rolling double occurs one sixth or 16.7% of the time. Not rare at all, almost mundane.
Republicans have the edge now, but this isn’t Obama 2012, when the blinkered couldn’t accept the results of multiple forecasters, all of whom had a consistent story.
Wang “wins” because his fundamental reasoning is sound: the polls are superior to models. In November. But what happens when you don’t have enough polls? Secondly, Wang has touted his superior accuracy in November, when polls are abundant and fundamentals modelling isn’t as useful. How accurate has Wang been compared to Silver and others in September? That’s the issue at hand here, since it is September, and Wang’s relying on sparse polling. Since he is choosing to be an outlier here, that means he won’t get away with having his model converge with everyone else’s in November and then exclaim that he beat Nate Silver. He is firmly saying that the Dems have a 75% chance of keeping the Senate. If he’s saying 30% in November, like everyone else, then he’s wrong.
Attempting to predict future performance of individuals based on past performance is actually a good example of faith over evidence. It’s like saying your favorite team will win because they have a track record of winning, even though half their players are injured. Or that your mutual fund manager will beat the S&P 500 in the future because he has in the past.
Most of the performance metrics I’ve seen have covered the accuracy of the last week before the election.
Put that aside. A sample size of 3 just isn’t sufficient to evaluate Sam Wang’s probabilistic model relative to other probabilistic models. If he says in August that the odds are 80% in favor of victory and somebody else says 60%, it would take a sample size higher than 3 to judge which model was better, even assuming that the respective models stayed constant from year to year. Which of course they didn’t: everybody adds tweaks.
The problem isn’t impossible though. I’d be open to looking at the August poll vs. fundamental evidence across a large sample of elections. Maybe such a paper has been written.
ETA: To be clear, I find his August claims “Plausible”, though not preferred. I find his claim that polls work better than fundamentals in May of an election year to be, “Surprising.” I’d like to see the evidence for that.
Of course relative “accuracy” of probabity statements of some event, be it election results or weather, requires a very high n. I’d call the weather service stating a 70% chance of rain to be highly accurate if it rained between 60 and 80% of the time they stated that. If it rained more 90% of the time after they said that then I’d consider it not to be very accurate, just as much as if it rained less than 50%.
Probably true that polls in May are not wonderful predictors for close races. The question however is whether or not polls even then are marginally better than throwng lots of punditry, I mean “fundamentals”, at the question instead? Not sure I know the answer either.
Silver doesn’t use punditry. He uses history. For example, the generic ballot shows Democrats ahead. Problem is, that’s one poll that even though it’s been done for decades, they still can’t get it right. Republicans historically outperform the generic ballot poll by +3.
The other example is the “bounce” candidates get when they win a primary. Incumbents always lead before we know who their opponents are, unless they are totally screwed rather than just probably screwed. If you aren’t accounting for that in May, then you end up shocked, as WAng has been apparently, by the tightening of those races. I remember people citing Wang talking about how strong Democratic incumbents were looking. Meanwhile, Silver and Cook and Sabato, etc. were accounting for the fact that the races would tighten. Wang didn’t see that coming.
What’s going to happen is that the polls will get better in the coming weeks and Wang’s model will be tweaked to be more in line with the other models. And it’ll end up doing very well and he’ll tell us how accurate he is. Just so long as you don’t read anything he writes before mid-October.
