A question for our polling mavens: When the poll’s quality is considered and then weighted, what does that mean exactly? To this non-pollster it sounds like, “Look, we know this poll is shitty, but not to worry. We assigned it a much lesser weight.” Isn’t the correct move for a low-quality poll to ignore it?
The whole concept seems bizarre. Two polls in the aggregator’s model could show opposite outcomes, two polls ostensibly assessing the same exact race. How does weighting and averaging produce anything valuable? One or both is dead fucking wrong. And for sure both cannot be right. “Well, we’ll mix them together. Problem solved!”
I’m sure I’m oversimplifying, and I get that well-run polls can still get things wrong. But I’m coming to the conclusion that the whole process is largely worthless. “But look, we can see after the fact that most of the polls were within acceptable statistical tolerances. And don’t forget, they don’t pretend to predict precisely.”
The point I see from your data is that the partisan skew of Republican pollsters is small compared to how far state level polls are usually off by.
A very small portion of pro-Trump respondents surely recognized the name of the polling organization and chose to participate because of that. That’s all it would take to generate these small differences. There is no need to hypothesize unethical pollster behavior.
You can think of the polling aggregators as putting polls in context. There is no right way to poll, so different groups will make different decisions and sometimes those decisions are will cause their results to skew. So one thing weighing does is say that a pollster is typically to the right/left of the average pollster. So a Trafalgar result of Trump +2 and Morning Consult +2 Harris might be equivalent, so they are treated as such. Weighing also takes account things like recency of poll, sample size, and pollsters track record. All polls are weighted for things not just the bad ones.
What to include is different between aggregators, but since we do not what the correct way to poll is really, people like Nate Silver tend to lean to include anything that has a reasonable approach to polling. If you know a firm is right of the average by three and their next poll is 5 points different, then that might tell you something even if the data may be skewed. Some others are a bit stricter, but the results are pretty similar across the board.
As for the value, while polls can be off, they are generally reasonable accurate. Certainly more accurate then listening to pundits or reading message boards
I think the urge to create special sauce just needlessly complicates. And the fact of the above, that adding the extra bits doesn’t change anything, demonstrates it is just no value added, or taken away.
Depends on how you’re defining “low-quality”. If a poll jumps all over the place (over time) and has no methodology listed or the methodology makes outside experts go “hmm”, the accuracy over time is questionable and it may be better to ignore.
But some of these we’re talking about are “decent method but mistargeted” (in other words, precise but not accurate). If you know a poll historically and consistently runs +2 versus past results (maybe because they use different but somewhat-reasonable and reviewable methods of extrapolating up to different population subgroups), then just subtract 2 every time before adding it into the average, and it’s still useful information.
This, coupled with the betting markets getting pushed to strongly favor Trump by singular whales keeps me pretty firmly entrenched in my belief that the relationship that people have with polling is entirely ass backwards.
The problem with their method is that polling was actually way wrong last election, so the “house effect” would be a lot less pronounced. That doesn’t make the polls any less fake this time, they just got lucky last time.
Yes, that one. The only comparable elections are those with Trump in them, the rest are meaningless. The point is the GOP leaning polls were more accurate, by pure luck.
I just think that Nate Silver and liberal Substack are talking past each other. Silver believes that if zone-flooding doesn’t affect the weighted polling averages (e.g. his own) by a sufficiently large number, then the zone-flooding doesn’t matter. Liberal Substack disagrees with Silver: “A gamed polling average” is just one of several potentially deleterious effects from zone-flooding – and probably not even the most serious. Reposting Jay Kuo for emphasis:
Some observers, such as Jon Favreau, have noted that these GOP-leaning polls haven’t been able to move the actual averages that much, maybe a point or so. But even that can be significant in a tight race. And besides, it’s more than just the averages. It’s the constant drumbeat of “Trump is winning” or “the race is essentially tied” headlines. They are affecting the media reporting while dampening the national mood with Democrats and raising expectations undeservedly with Republicans. As I’ll discuss later, that all matters a lot.
Thing is the statement that “the race is essentially tied” by polling IS ACCURATE whether one looks at only the few highest select polls or the aggregates. The substacks keep trying to imply and sometimes outright claim that such is not the case BUT IT IS.
Of course there may be, likely will be, a systemic error, and hopefully that error will favor Harris. I think it will. But by polling, even only looking at the few select best, it is essentially tied.
Now can media, will media, cherry pick to create stories that cause anxiety and drive clicks? OF COURSE. Always. But that is true if the aggregators list polls or do not and has nothing to do with statements that the aggregators are gamed.
Yes in the silos of these substacks these claims are not in dispute. Just like the claim of Trump actually won in 2020 is not in dispute in other silos.
You have yet to show any evidence that this is true. These polls have no effect on the aggregate effect. They have minimal effect even if you were doing a straight average of polls. They haven’t remotely changed the narrative of this race. Voter behavior hasn’t been effected. It is a tossup anyway you look at it.
If you want to argue that these pollsters are trying and failing to achieve these things that is a little more defensible I guess. I still don’t buy it, but you at least would have to dismiss away reality.
So you want to use two data points, one of which was during Covid. That seems like a pretty meaningless sample size to me. Election data is going to be subject to error because the data is generally lacking and things do change, but I think you have a better chance of accuracy if you look at the bigger picture.
I’m not sure anybody truly answered this question (assuming it’s a sincere question). It has to do with the errors associated with random sampling.
If there is a bag with millions of balls in it, some percentage of which are red, some blue, and some various other colors. You have a handful of people reaching in and randomly pulling out a small handful of balls and reporting what colors they get.
Clearly each person is going to get a different answer of red/blue/other. Some of them will be closer to the correct percentage than others. But, mathematically at least, the most correct answer is almost certainly going to be averaging all of the various measured rates. Simply by having a larger overall sample you should be reducing sampling error.
Now polling for an election is obviously more complicated. First, you have the problem that the balls don’t all “want” to be pulled out of the bag at the same rate - you can only poll people that answer the phone/text/whatever, and that’s an extremely small percent, and typically not very representative of the whole bag. Pollsters try to handle this by weighting by demographics and, more controversially, by past recalled vote (i.e. trying to balance the folks that claim to have voted for Trump vs. claim to have voted for Biden to match what actually happened in 2020).
Second, you also aren’t exactly trying to answer the question of “what percentage of the balls in the bag are red/blue/other?”. It’s actually “what percentage of the balls that will actually cast ballots are red/blue/other?”. All pollsters have slightly different “likely voter” models that are less or more restrictive in how they handle respondents that aren’t sure they will vote.
It’s probably a fair question to ask whether averaging polls with different weighting and likely voter methodologies actually improves results. I think the answer is probably yes, in that we can’t know ahead of time which methodology will be correct so it would be wrong to assume one way or the other.
Separately, I would give this whole “GOP-aligned pollsters are trying to game the averages” line a whole lot more credit if there was a sea of “non-GOP pollsters” showing a blowout or something like that.
In that scenario it would possibly make sense to try to craft a narrative to show my guy was still competitive just so supporters didn’t jump ship or decide not to show up.
But that wave of great Harris numbers just doesn’t exist. Look at Michigan - there isn’t a single poll in October that shows Harris ahead by more than three points. And there is a Quinnipiac poll (not GOP-aligned, I think you would agree) that has Trump up 3.
PA is similar, as is Wisconsin.
There is just no combination of polls you can pick out, even with some extreme cherry picking, that paints Harris as anything better than a 60% favorite.
On liberal message boards like this one, and similarly-aligned Substacks and social media: Trump is toast and the polling just isn’t capturing how badly he’s going to lose. This is almost all based on Trump’s characters, actions as President, and factors other than polling data.
In the “mainstream” media: this election will be extremely close just like Trump’s previous two elections. Harris will likely win the national popular vote by a few percentage points, but the Electoral College will come down to some combination of the rust belt (MI/PA/WI) and the sun belt (AZ/NV/GA/NC). Polling data backs this up both in it’s stability over time and it’s general alignment with past results.
On right-wing boards: Trump will cruise to election because polls consistently underestimate his support. In 2016 and 2020 he overperformed his polls, because he has tons of support with the kinds of folks that don’t answer polls. The fact that polls show it as being close confirms that he is going to win (oh, and if he doesn’t that’s because the Democrats cheated and used these fake polls to back up their steal).
I really struggle to see how the current polling data we have, be it “red-wave” or not, changes any of those narratives.
To an audience only casually familiar with polling (e.g. a heckuva lot of people), consistent positive polling – even +1s and +2s – is as good as being up a cool +5 or +6. Being able to say “So-and-so is UP in the battleground states!” is the thing – being up “in the battlegrounds” by +0.5 or +5 are essentially the same.