Spotted the TV weatherman
I mean, if a model gives one candidate a 97% chance of winning, and attributes the remaining 3% to the possibility of an unanticipatable catastrophe like a medical emergency, and then that candidate loses in a fairly conventional election, you might be justified in calling that forecast “wrong”. But 30% chances happen pretty frequently. Silver’s forecast was basically saying “If you flip two coins and they both come up tails, then Trump wins”. Which, yeah, favors Clinton, but you don’t want to stake the fate of the free world on two coin flips.
These comments are spot on.
At the point in time 538 weights polling data very low. It is essentially a WAG pontification that the facts of the economy (over how people perceive it) and incumbency will determine how people vote and that we should la-la-la the polls. Mostly.
Could be correct. I don’t know. But there’s no solid evidence to back it up, it only gets there by mostly ignoring the polls, and its approach leads to conclusions significantly different than the other major models (including the original 538 one that Silver still uses). It is new, it is the outlier, and it is more opinion than data.
I think that’s unfair. His position on COVID, AIUI, is that by late 2020 it was apparent that COVID rates weren’t any lower in the parts of the country that were scrupulously masking and social distancing than in the parts that were ignoring all that stuff. Therefore, we could have saved ourselves a lot of economic damage by abandoning those ineffective anti-COVID measures, and our failure to do so was largely because most people’s opinions on the issue were driven by partisanship more than science. Today, on the other hand, there are significant differences in COVID rates between red and blue States, and that’s because vaccines, unlike masks, actually work.
I’m not familiar enough with the epidemiology to say if he’s right or not, but it’s not a crazy position. He’s not denying that COVID is real, or that masks and social distancing mandates were reasonable measures to take based on what was known in early 2020, or that everyone should get fully vaxxed and boosted. He’s not claiming that the alleged policy failures were due to any kind of conspiracy, just to ordinary human foolishness.
As for his skills as a pundit, well, he and Jon Stewart were among the first to sound the “Biden is too old” alarm several months ago, so that’s a big win. One thing I like about him, though, is that he’s always careful to draw a bright line between “things I believe based on my special expertise as a statistical analyst” and “my own personal WAGs”, and he knows that his audience is mostly interested in the former.
Technically, you can’t say that a single probabilistic forecast is ever “wrong”. But if someone, like Nate Silver, makes thousands of probabilistic forecasts over a period of years, you can certainly judge their accuracy. If he’s judging ten uncorrelated events to each have a 97% chance of occurring, and only five of them do in fact occur, you can reasonably say it’s extremely unlikely that he is a generally accurate forecaster. By this standard, Silver’s record is excellent. (I mean, I’m taking his word for that, I haven’t personally checked, but the data is all in the public record, I assume someone would have pointed it out by now if he were lying).
I believe he analyzed it at some point. If I recall his probabilities were a little low. Like if something should have happened 80% of the time it actually happened more like 90% of the time. However given how highly correlated election results are is hard to draw too many conclusions from a handful of elections.
As a pundit Silver is just another talking head. No better no worse. I do respect his data analysis.
So, Nate Silver’s election methodology produces what is essentially a compound Bayesian inference “likelihood” model; that is, it uses available information and informed guesses to produce a series of weighted estimators (of which polls are one significant data set) forming an ad hoc distribution (“priors” or a prior distribution in Bayesian terminology). This is a methodology that is heavily used in data science because it has great predictive power when applied to a large dataset compared to frequentist methods, and especially if it is progressively refined by updating the model with post hoc observations (“posteriors”) which form the next round of priors. However, because this election model is making predictions (point estimates) about singular events like the binary outcomes of elections it isn’t really meaningful to speak of whether such a prediction is “right” or “accurate”, but rather if it is useful; that is, that it is reliably indicating a trend rather than making any kind of verifiable prediction about an absolute outcome, and in a larger perspective if the methodology in general is providing a good estimate of electoral trends (which it empirically has).
The general way of evaluating this kind of model used is to see what how significant differences are between priors and posteriors when the model is updated. This, again, can’t evaluate how “accurate” the model is, but it does demonstrate that the model is perturbative only within acceptable limits and is not prone to wild swings with small changes in input parameters. That Silver’s model estimate a 30% likelihood of Donald Trump winning the 2016 election is not, as @Whack-a-Mole (and others elsewhere) have claimed, an indication that the model was flawed but actually demonstrated reasonable predictive power given that no other credible estimates gave Trump anywhere close to that likelihood of winning the election. Frankly if there is any criticism of the model it is that it likely relied too much on polls that were biased against Trump, skewing the estimated likelihood lower for Trump than it probably should have been.
Polls themselves are only really meaningful in terms of being a sampling of what the election would look like if it were conducted at the time the poll is taken, and you absolutely cannot make a prediction about how an election will turn out from polls that are months away from election day. However, the trends that are indicated by good polling will provide a useful indicator of how a candidate’s policy and persona are accepted or rejected by the public. That Hillary Clinton, for instance, was seen in a largely negative light, and the bumps she would get in polling after some positive event would be followed by a regression back to that fairly negative baseline should have been quite worrying to the Clinton campaign, especially when there were occurring in critical swing states that her campaign largely ignored. That Trump’s net approval, while also negative, never really stayed down even following brief dips after revelations like the “grab them by the pussy” video, should also have been an indication that he had a robust voter base that wasn’t going to be swung by transient events.
That the current 538 (not “FiveThirtyEight”, which was Silver’s original branding of the site) model is highly reliant upon historical predictors and “economic fundamentals” (which should be as suspect as the unduly rosy economic indicators themselves are) to give him a positive position should be highly concerning because the economy in terms of what many voters are experiencing is not all that great (especially in the last eight to ten months), and the “incumbency advantage” gets inverted when that happens under a candidate who is also a sitting president; just ask Jimmy Carter or George H.W. Bush about the privileged position their incumbency offered. Even without much of the Democratic base and (behind closed doors) party leadership questioning Biden’s mental solvency after that disastrous debate appearance, Biden would be facing a somewhat uphill challenge. And while you’d think conviction and various other perfidies would be hurting Trump as much or worse, for his base they are actually a badge of honor, an achievement that shows he is an outsider even after having presided over what is inarguably the worst presidential administration in a century. The logic that Biden has and can maintain the advantage, absent of some really robust polling data, is tortured and not believable.
Stranger
I’d say it’s even less. Weather is a natural unconscious process. But an election campaign is a guided conscious process. They are run by people who are aware that Election Day is happening on November 5 and have scheduled the campaign accordingly. It’s like saying “If Thanksgiving happened today, do you have a turkey and cranberry sauce ready?” A turkey that’s sitting in my fridge today will be spoiled by Thanksgiving. It’s the guy who buys a turkey around November 21 who will have one ready to eat on Thanksgiving day.
It’s handy for getting an idea of how the electorate is feeling, but I agree, it’s only of limited value and best looked at as part of a trend.
The predictive value of polls certainly increases as Election Day gets closer, but there’s no magic point at which they go from worthless to significant. Even a couple years out from the election, candidates leading in the polls go on to win somewhat more often than people who aren’t.
And based on past history, you can estimate how safe you should feel based on polls at any given time in the election cycle.
Just like in football, for example, if you turn the game on and see your team is 14 points down in the first quarter, you’re probably going to lose, but (assuming the teams are evenly matched), it’s certainly not hopeless; you can remember seeing a lot of games where teams come back from deficits like that. If you see that you’re 14 points down with three minutes to go, you’re almost certainly screwed.
And then you have to decide how relevant past history is to the particular case you’re looking at. This is the part that’s art rather than science. Continuing with the football analogy (which I cribbed from Silver), if you’re 14 points down in the first quarter, and your star quarterback is being carried off the field on a stretcher, your chances of winning are dramatically less than you would expect without knowing that information.
Whatever one thinks of the current version of 538 … at least they’re not standing still. Trump now leads by the slimmest of margins – 501 simulations to 496 for Biden.
Seems apparent that “pressure for Biden to drop out” is being weighed in somehow.
Sure, but to me there are two big red flags with calling what 538 is putting out a “polling model”:
- It doesn’t seem to respond to changes in polling data much, if at all. Raw polling averages have shown at least a 2% swing towards Trump since the debate, but the prediction at 538 hasn’t moved at all. That is odd.
- It seems to combine fundamentals and polling averages in an odd way. Look at Wisconsin (Who Is Favored To Win The 2024 Presidential Election? - Wisconsin | FiveThirtyEight). It has the election-day polling as Trump +2.4%. It has fundamentals as Biden +0.1%. Then it has a “combined” result of Biden +0.9%. That doesn’t make any sense, and demands an explanation.
- It’s tails seem far too large to be useful. It shows Biden with a 13% chance of winning the popular vote by over 10%. Does that smell right to you? It shows a combined 35% chance that one of the two candidates gets over 350EV. Really?
It feels like either there are some bugs (not impossible - it’s an entirely new model) or some adjustment is being made during the “combine polls and fundamentals” step that assumes Trump is over-represented in the polling.
I notice the wording (which I don’t’ remember from before): “Our final forecast of the popular vote, based on both polls and fundamentals and accounting for the chance that polls systematically underestimate one candidate. Before Election Day, the final forecast in some states can be more Democratic or Republican than the fundamentals and polls because of patterns of overperformance in similar states.” (emphasis mine).
To me that reeks of “Democrats have over-peformed in special elections this year so we’re going to make an adjustment assuming that will be true in the fall”.
Except it seems like Morris is saying the score itself is wrong. That since he’s seen Democrats come back from being down a lot over the last two years or so (winning special elections) that when the polls say they are down they really aren’t.
That may be true, but it’s not really science IMO. It brings back horrible flashbacks to the “poll unskewers” from the Obama/Romney race. Assuming that special election results tell you that general presedential polls are inaccurate doesn’t make any sense, especially when we have a pretty strong prior of 2020 which showed that Trump actually over-performed his polling against Biden.
I’m not sure Morris is talking about special elections, and it would indeed be BS if he is.
He could just be talking about the entirely defensible practice of “Almost nobody is bothering to poll small, non-swingy State X, so there’s not much data there. However, it’s demographically very similar to its neighbor, big State Y, where there’s a lot of polling. So we can improve the accuracy of our forecasts by assuming that the vote movement in State X mirrors that in State Y”.
You know, just because the founder is having a hissy fit about the company he SOLD, that does not make this true:
538 is No Longer a Credible Source.
That headline is false.
Perhaps a better way for the OP to have said it is something like
In Silver’s opinion, 538 should not be releasing any predictions while it’s own management agrees their model cannot validly include current polling data. Which they predict will be the case until along about September. Until then Silver tells us they’re not reliable now because their own management has said as much.
Kinda long for a title. But less misleading.
With a sufficient sample size, you can compare probabilistic model prediction with outcome (as you noted later upthread).
You can also evaluate a probabilistic model by looking at its underlying assumptions Morris’s attempt apparently assumes that polls are worth very little 75+ days away from the election. That would be around Aug 20th. Silver’s model assumed the same at 160+ days in 2020 IIRC.
So which is more accurate? Does Silver cite academic articles saying one or the other? Does he present statistical evidence? Morris’ take seems to be that the race only takes shape around Labor Day and afterwards. That’s not obviously false.
I don’t know, when a model has Biden favored more in the combination of polls and fundamentals than in either one separately (as documented by @Jas09 ), and there’s no explanation for why or how that’s happening, I think that’s enough to say that it’s not a credible source.
To continue with the football analogy, it’s a bit like saying that Tim Tebow was a good pro quarterback because the Broncos won some unlikely games when he was there. Anyone who watched the games could see that it wasn’t Tebow, it was mostly the defense and some luck and if anything Tebow was making games harder to win. So yes, the Democrats have managed to over perform some in 2018 and 2022 but they weren’t great wins. They were the defense bailing out the offense. Eventually the offense will be so unproductive that the defense simply cannot be effective.