538 Nowcast vs PEC Snapshot

Politics & Elections
1

There are many differences between Nate Silver’s approach and Sam Wang’s. Silver applies weightings to polls based on the pollster’s historic accuracy and how sound he thinks their methodology is. He adjusts for pollsters’ “house effects”. He projects older polls forward in time based on a trendline adjustment. He combines each state’s polling average with regression estimates based on region and demographics, giving more weight to the regression in states with less polling. So far as I know, Wang does none of these things, pretty much taking the polls as is. (I don’t mean this as a criticism of Wang – there are certainly some reasons to favor simplicity.)

However, I’d like to mostly ignore those differences and focus on something else – how they estimate uncertainty. Wang starts with an estimate of where things stand today that has very little uncertainty, and then assumes random drift between now and election day, the magnitude of which is based on historical data. He also applies a Bayesian prior based on the long-term state of the race. Silver runs simulations with several random errors added in: one representing a systematic error in the polling nationwide, others representing systematic error in the polling in particular regions or states with particular demographics, and others representing systematic error in the polling of a particular state. The magnitude of these errors is based on the time until the election plus various other factors like the number of undecided voters and the size of the state’s population.

But even on election day, the error terms in Silver’s model can be far from zero. We can see this from the Now-cast, which sets the date of the election to today, and currently shows Clinton with only a 67% chance of victory despite having her ahead nationally by 3.5 percentage points.

Wang’s snapshot also addresses “who would win an election held today”, and shows Clinton with something like a 99% chance of victory. Wang deemphasizes this percentage (he doesn’t state it explicitly, but notes you can get it by summing the histogram), because after all the election isn’t today and he presumably doesn’t want to give the false impression that he’s saying Trump has essentially no chance. Likewise, Silver doesn’t treat the Now-cast as a prediction.

That said, I think the difference is important, because it tells me something about how much I should trust the predictions they will make once the week of the election arrives. At that point, there’s almost no time left for drift, meaning Wang’s prediction and snapshot should converge, and likewise Silver’s Now-cast and Polls-Only model should converge. (His Polls-Plus model, which considers fundamentals, should also converge to the others, since the weight it places on fundamentals goes to zero as the election approaches.)

So who to trust? Is it really reasonable to think that (given no time left for drift) the polls give us 99% certainty of the outcome? On the other hand, is it really reasonable to think that if one candidate is down 3.5 points nationwide in the polling aggregate come election day, they still have a 1 in 3 chance of victory?

(DSeid, I know we had some back and forth on this in a previous thread. I appreciated your take on things. Please don’t think I ignored it just because I’m bringing it up again for further discussion.)

2

As someone who really isn’t qualified to talk about the math - I just look at it in general terms. Both men have similar, highly successful, track records. Both methods show that Clinton is winning. But they’re using different tools, so there’s no reason to expect that they will get identical results.

The fact that two different methods agree on the outcome, even if their numbers are different, is a factor in trusting their results.

Also, fwiw, I’d like to point out that Wang’s method includes only state polls, not national polls. Silver includes both. That accounts for some of the difference.

I trust them both, that they know what they’re doing, and I’m confident that they could show their work if asked for it.

3

Silver himself really doesn’t like the NowCast.

4

Indeed. And to be clear, I don’t expect the NowCast nor Wang’s snapshot to be an accurate prediction – neither is intended to be a prediction at all.

But if I understand correctly, the methodology used today for the Now-Cast and the Snapshot is the same as the methodology used on election day for the 538 Polls-Only prediction and the PEC prediction, respectively. So the fact that they differ by so much in terms of the reported uncertainty (despite both having Clinton up something like 3.5 or 4 points) suggests that if the election is still this close in November, the actual predictions would have drastically different uncertainties.

We could talk about the actual predictions as they stand today, but then that brings in additional questions like whether Wang’s use of random drift makes more or less sense than Silver’s adjustments to his random errors. I’d rather focus on a single question: Which model (if either) get’s the short term error right? By “short term”, I mean something like the difference between what we know on the morning of election day to what we know that night.

Nate’s model seems to be saying “If on election day the election is exactly as close as it is today, then there’s a 1 in 3 chance we’ll get the outcome wrong”, while Wang’s seems to be saying there’d be more like a 1 in 100 chance he’d get the outcome wrong. Given that they’re both starting from the same information, that seems hard to reconcile. It seems like they fundamentally disagree about how much you can trust the polls to tell you who would win an election held today.

5

And I realize this is all somewhat subjective, but I think there’s a factual question in here as well: How often does an aggregate polling lead of about 4 points on the morning of the election end up holding up? (Although, U.S. Presidential Elections being fairly rare events, I’m not sure we have the data to answer that.) If it’s far less than 99% of the time, or far more than 67% of the time, then something seems fishy here.

6

As for your last question, 1948 (“Dewey Defeats Truman”) is the clearest example, though polling wasn’t as sophisticated then. But that implies we get it that wrong about 1 out of every 15 elections – in other words, a percentage right between the Wang and Silver numbers you cited.

7

P.S. I suggest we call the “Dewey Defeats Truman” scenario a November Surprise. :slight_smile:

8

I was under the impression that in the last 16 elections popular vote matched up with polling with the caveat that Gore didn’t actually win the presidency. So that would mean a) 1 out of 17 have been wrong and b) that 1 caused a massive rethink of polling methodology.

9

Well we can at least look at the last several elections and see how close the final RCP polling average was to the final result even if the difference did not impact who won.

Romney v Obama it called Obama as the winner but it was off by 3.2, underestimating his margin.

McCain v Obama it was within 0.3

Bush v Kerry RCP average within 1.5

No RCP average for Bush v Gore but looking at the final sets of polls and using the last five the aggregated polls were calling it pretty close to tied.

So at least in recent history pretty good but off by 3.2 has occurred so off by 4 cannot be considered as impossible.

10

Yes, Dewey Defeats Truman was only possible because of inherent flaws in how polling was conducted at the time, flaws which the pollsters then fixed. But the point is, our current polling could also have fundamental flaws, and we have no way of knowing until we see the actual results. It’s unlikely, but it could happen. How unlikely? We don’t know, but I don’t think we can do any better in estimating it than to look at how often in the past there have been fundamental flaws in polling.

11

The decline of landline phones could be a factor but there is apparently enough randomness among those who don’t respond that it doesn’t appear to skew polling. However, the inclusion of two higher-than-usual profile 3rd party candidates, voter suppression efforts, and unpopularity of both candidates will affect actual voters showing up and casting actual votes. I suspect the hard part will be predicting the turnout.

I suspect that Trump is going to have a hell of a time winning Florida, Nevada, Virginia, and New Hampshire, which is why a lot of people are probably already cautiously optimistic that Hillary will win. If Trump were to lose these states, then that would mean that in order to win, he would have to win Wisconsin, Ohio, Pennsylvania, and Michigan. But here’s the thing: I think his current campaign management team is now aware of this and they’re heading in that direction. They’re going to try to win the white votes in these states. It’s a long shot, but I think it’s actually doable.

12

For the past three elections (2012, 2008, 2004), how close were 538 vs PEC in their estimates on how many Electoral Votes the candidates would get?

I assume both correctly predicted the winner in the past three elections, but seeing who came closest in terms of EV’s can show us whose model is more accurate.

From their websites I found the following:



Number of Electoral Votes that the winner of the election got 
or was predicted to get as of Election Day:

       Actual    538   PEC  
------------------------------
2012     332     313   303  
2008     365     349   352  
2004     286     N/A   286

In 2012 538 was closer (19 vs 29 difference), in 2008 PEC was closer (13 vs 16), and in 2004 PEC was spot on while 538 has no data for that year.

Overall similar. If we had 538’s estimate for 2004 we would be able to make a better comparison.

13

From my memory, 538 picked every single state correctly in 2012. Where did you get those numbers?

14

Who is more accurate right before election day is meaningless, since both use polling. Silver however projects into the future with what he calls a ‘special sauce’ and Wang doesn’t. That had Wang calling the senate for the Democrats until nearly October 2014, and Silver seeing the coming Republican wave possibility much earlier.

What we want to know is who is more accurate NOW, in September, and Silver’s record is clearly superior.

15

On Sep 5, Sam Wang rated the GOP’s chances of taking the Senate at 25%. Silver had them at 64%.

Silver wins in a landslide.

16

Calling the EV correctly is likely not the best measure for judging model accuracy … the difference between models that both correctly called a state a win for candidate X who then wins it by a margin of 3 but one says X wins by 0.5 and ranks it a 53% probability and the other who says the margin will be 3 and ranks it as 80% is meaningful, even if the EV impact is the same.

That’s where Brier scores come in, which factor in confidence as well.

In 2012 Drew Lizner actually did best, followed by Wang, then Silver, and a bit farther back the betting markets. Wang “crushed” Silver on the 2012 Senate races on that metric as well. (And did not do as well in the 2014 mid-terms.)

Wang also beat out Silver on that measure in 2008.

Overall Wang does state more confidently and is usually correct when he does, with one very notable failure that Silver made a lot of hay over (the Sharon Angle call).

As for the place of “fundamentals” in the mix … that is the question. When there is little polling data, very early in the race, then they likely add lots of value. But once the polling gets denser I see no evidence that fundamentals adds much. Silver even is not completely convinced they do either which is why he offers three models up, one of which is fundamentals heavy (polls plus) to one completely fundamentals-free (nowcast).

One can utilize a Bayesian prior that is polling based instead of fundamentals based - that’s Wang’s approach.

17

But the discussion is who is more likely to be right now, not who is going to have the more accurate prediction on Nov. 7. Right now there’s a lot of gnashing of teeth over Silver’s more pessimistic view of Clinton’s chances vs. Wang’s. Who has the better record in September?

18

I have a hard time believing that Nate’s nowcast is as advertsied. That is, that it’s Nate’s best estimate of what would happen if the election were held tomorrow. If you scroll down the now cast page to “Who’s winning the popular vote” you see the following sentence:

Although they are centered differently, the width of the error distribution look identical between the Now Cast and the Polls only. This suggests to me that even the Now Cast considers that there are still two months left and that the polls may be very different then. Otherwise we would expect much narrower error bars in the Now cast than in the Polls only Cast.

I think the real difference between Nate’s nowcast and say the Polls only is that the Now Cast centers the vote according to the most recent polls. Basically suggesting that the current state of the race right this instant represents the most likely state of the race in 2 months. The Polls only forecast considers a longer term and discounts short term changes that might represent bumps from the current news cycle and suggests that the final poll numbers are more likely to match the long term status quo of the election than they are today’s numbers.

19

Honestly, I think lumping Virginia with Florida is really comparing two horses of a different color. Virginia is very close to being a true blue state. Florida looks like a tossup and has never been firmly in the Clinton count.

20

Do we know how the percentage of undecideds at this point of the election compares to prior elections? I’m curious if this election is an outlier in terms of the number of people who haven’t made up their minds yet, which would I think make the polling less accurate.