# So will anybody ever trust the polls again? **URL:** \ **Category:** Politics & Elections\ **Created:** [November 4, 2020, 5:49am UTC](https://boards.straightdope.com/t/so-will-anybody-ever-trust-the-polls-again/924714 "2020-11-04T05:49:40Z")\ **Posts on this page:** 20\ **Page:** 13
**Author:** ![Lance\_Turbo](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/lance_turbo/32/6156_2.png) [@Lance\_Turbo](https://boards.straightdope.com/u/Lance_Turbo)\ **Post date:** [November 10, 2020, 5:13am UTC](https://boards.straightdope.com/t/so-will-anybody-ever-trust-the-polls-again/924714/245 "2020-11-10T05:13:00Z")
> [@Pleonast](#): > > I’m curious how much of a difference it’d make in the scores of all the long-shots were included. I wasn’t curious about this at all. Never considered it and didn’t collect data on it. You can see in their data that 538 didn’t either. ---
**Author:** ![DMC](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/dmc/32/18049_2.png) [@DMC](https://boards.straightdope.com/u/DMC)\ **Post date:** [November 10, 2020, 5:14am UTC](https://boards.straightdope.com/t/so-will-anybody-ever-trust-the-polls-again/924714/246 "2020-11-10T05:14:40Z")
> [@Lance\_Turbo](#): > > Links to data sets… I appreciate it, but your predictit data is missing entity data. [The electoral college](https://en.wikipedia.org/wiki/Electoral_college). 🤨 ---
**Author:** ![Lance\_Turbo](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/lance_turbo/32/6156_2.png) [@Lance\_Turbo](https://boards.straightdope.com/u/Lance_Turbo)\ **Post date:** [November 10, 2020, 5:18am UTC](https://boards.straightdope.com/t/so-will-anybody-ever-trust-the-polls-again/924714/247 "2020-11-10T05:18:23Z")
Argh. I just (too) quickly wrote a script to smash the 56 data sets into one data set to post and I left off a column. The transformed PredictIt data does have that info though. ---
**Author:** ![Pleonast](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/pleonast/32/1183_2.png) [@Pleonast](https://boards.straightdope.com/u/Pleonast)\ **Post date:** [November 10, 2020, 5:19am UTC](https://boards.straightdope.com/t/so-will-anybody-ever-trust-the-polls-again/924714/248 "2020-11-10T05:19:25Z")
> [@Pleonast](#): > > I’m curious how much of a difference it’d make in the scores of all the long-shots were included.   Let’s try an example. Guess 1: Biden 70%, Trump 30%, Longshot 0% = 24.49% rms error. Guess 2: Biden 69.5%, Trump 29.5%, Longshot 1% = 24.51% rms error. Guess 3: Biden 70%, Trump 29%, Longshot 1% = 24.10% rms error. Guess 4: Biden 69%, Trump 30%, Longshot 1% = 24.91% rms error. So, yeah, not a big difference. But I think it’s more fair to include them. ---
**Author:** ![Lance\_Turbo](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/lance_turbo/32/6156_2.png) [@Lance\_Turbo](https://boards.straightdope.com/u/Lance_Turbo)\ **Post date:** [November 10, 2020, 5:20am UTC](https://boards.straightdope.com/t/so-will-anybody-ever-trust-the-polls-again/924714/249 "2020-11-10T05:20:43Z")
> [@Pleonast](#): > > So, yeah, not a big difference. But I think it’s more fair to include them. But no 3rd party candidate had a win probability even as high as 1%. ---
**Author:** ![DMC](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/dmc/32/18049_2.png) [@DMC](https://boards.straightdope.com/u/DMC)\ **Post date:** [November 10, 2020, 5:22am UTC](https://boards.straightdope.com/t/so-will-anybody-ever-trust-the-polls-again/924714/250 "2020-11-10T05:22:58Z")
> [@Mijin](#): > > I disagree. It’s like saying that the last two coin flips were heads so maybe it’s a biased coin? Yes, it’s just like saying that, except this isn’t a random coin that he got from the bank, but a coin that he has never seen and was simply told the outcome of the flips. He’s asking to see the coin. > [@Mijin](#): > > Because firstly, I didn’t say anything about only considering these two elections. The prediction markets have been producing good results (not always, but compared to other methods like polls) for many years now. There’s plenty of data to help us ascertain how often prediction markets correctly predict outcomes. And he (and I) have been asking for the data that backs that up. @Lance_Turbo hasn’t provided it and I don’t expect him to. Unless they expose it clearly for all to see, it’s a pain to gather. Based on what he stated earlier, it might even be impossible to gather after the fact from PredictIt, for example. > [@Mijin](#): > > Because secondly, if we were insisting on putting on blinkers and looking at just the Trump election(s), the answer to that question would be basically unknowable. How could we rule out a net bias for a system that fundamentally is the aggregation of individual bias? We can’t. Who is insisting on the blinkers (blinders?), because I don’t believe that it’s me or @Fotheringay-Phipps. I’d like to see a shit ton of elections, not just this particular presidential election. Again, in case you think I’m simply parroting Fotheringay-Phipps, out of 100 random threads in this forum or Great Debates, I’m liable to side with Fotheringay-Phipps 1 time and Lance\_Turbo 99 times. I’m pretty sure Lance\_Turbo would back me up on that claim. ---
**Author:** ![Lance\_Turbo](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/lance_turbo/32/6156_2.png) [@Lance\_Turbo](https://boards.straightdope.com/u/Lance_Turbo)\ **Post date:** [November 10, 2020, 5:27am UTC](https://boards.straightdope.com/t/so-will-anybody-ever-trust-the-polls-again/924714/251 "2020-11-10T05:27:49Z")
Corrected… PredictIt Raw: [PredictIt\_raw.csv - Google Drive](https://drive.google.com/file/d/1K0jafsh4jN6z_qCuAIdQzJK4BmIrWzlb/view?usp=sharing) Link in previous post is probably dead now. ---
**Author:** ![Pleonast](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/pleonast/32/1183_2.png) [@Pleonast](https://boards.straightdope.com/u/Pleonast)\ **Post date:** [November 10, 2020, 5:28am UTC](https://boards.straightdope.com/t/so-will-anybody-ever-trust-the-polls-again/924714/252 "2020-11-10T05:28:42Z")
> [@Lance\_Turbo](#): > > But no 3rd party candidate had a win probability even as high as 1%. Even after normalizing so the sum of all candidates is 100%? ---
**Author:** ![DMC](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/dmc/32/18049_2.png) [@DMC](https://boards.straightdope.com/u/DMC)\ **Post date:** [November 10, 2020, 5:29am UTC](https://boards.straightdope.com/t/so-will-anybody-ever-trust-the-polls-again/924714/253 "2020-11-10T05:29:26Z")
You know how we nitpicky people like our data, raw! It’s all good, but I was somewhat curious to dig into the favorite/longshot bias a bit. Again, not understanding betting markets fully, I would assume that if someone over or under pays, that someone with better data would immediately take advantage, thus pushing the price back to where it belongs. While I can certainly see favorite/longshot bias being an issue a decade ago, with all of the fantasy betting and everything else going on, I would guess that many bettors use models these days, thus reducing the bias. Again, purely hypothetical, so I wanted to play around a bit. ---
**Author:** ![DMC](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/dmc/32/18049_2.png) [@DMC](https://boards.straightdope.com/u/DMC)\ **Post date:** [November 10, 2020, 5:33am UTC](https://boards.straightdope.com/t/so-will-anybody-ever-trust-the-polls-again/924714/254 "2020-11-10T05:33:13Z")
Thank you kindly, I’ve downloaded all three datasets. Damn, that 538 file needs a data dictionary bad. I’ll try to hunt one down. ---
**Author:** ![Lance\_Turbo](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/lance_turbo/32/6156_2.png) [@Lance\_Turbo](https://boards.straightdope.com/u/Lance_Turbo)\ **Post date:** [November 10, 2020, 5:35am UTC](https://boards.straightdope.com/t/so-will-anybody-ever-trust-the-polls-again/924714/255 "2020-11-10T05:35:05Z")
> [@Pleonast](#): > > Even after normalizing so the sum of all candidates is 100%? At no point, in any state, did any 3rd party candidate have other than an infinitesimal chance of winning an electoral vote in the 2020 election. ---
**Author:** ![Pleonast](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/pleonast/32/1183_2.png) [@Pleonast](https://boards.straightdope.com/u/Pleonast)\ **Post date:** [November 10, 2020, 5:35am UTC](https://boards.straightdope.com/t/so-will-anybody-ever-trust-the-polls-again/924714/256 "2020-11-10T05:35:12Z")
To be clear, l like your approach, **Lance\_Turbo**. I’m poking at the long shots, because it’s a clear difference between 538, which gives exactly 0% chance to them, and betting markets which give them long odds. I think it’s unfair to the betting markets to exclude that. ---
**Author:** ![Mijin](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/mijin/32/9369_2.png) [@Mijin](https://boards.straightdope.com/u/Mijin)\ **Post date:** [November 10, 2020, 9:34am UTC](https://boards.straightdope.com/t/so-will-anybody-ever-trust-the-polls-again/924714/257 "2020-11-10T09:34:31Z")
> [@DMC](#): > > Yes, it’s just like saying that, except this isn’t a random coin that he got from the bank, but a coin that he has never seen and was simply told the outcome of the flips. He’s asking to see the coin. No, it’s much more than just that. Fotheringay-Phipps has said that he “strongly suspects” there is bias towards Trump, “for the reasons given”…where the “reasons given” are just these two, more accurate, predictions. These suspicions should be backed up with something, and the fact he can’t do that is why he finally ‘agreed to disagree’ > [@DMC](#): > > And he (and I) have been asking for the data that backs that up. I don’t know the details and didn’t claim to. I only know that in various summaries e.g. by The Economist and the Atlantic comparing prediction markets (such as the IEM) to polls, the former has generally performed much better in the last 3 elections plus 1988 to 2000. I’m not sure about the '04 and '08 elections. Economist and Atlantic are behind paywalls, but [this](https://www.researchgate.net/publication/282123487_PREDICTION_MARKETS_VS_POLLS_-_AN_EXAMINATION_OF_ACCURACY_FOR_THE_2008_AND_2012_US_ELECTIONS) is an example of the kind of comparison I mean. ---
**Author:** ![naita](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/naita/32/5862_2.png) [@naita](https://boards.straightdope.com/u/naita)\ **Post date:** [November 10, 2020, 2:09pm UTC](https://boards.straightdope.com/t/so-will-anybody-ever-trust-the-polls-again/924714/258 "2020-11-10T14:09:34Z")
If I predict five times that there’s 50% chance of heads after a coin flip, and Lance\_Turbo predicts five times that there’s a 70% percent chance, his approach will lead him to claim he was more accurate. I say he’s using an inappropriate comparator. ---
**Author:** ![Monocracy](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/monocracy/32/4493_2.png) [@Monocracy](https://boards.straightdope.com/u/Monocracy)\ **Post date:** [November 10, 2020, 2:09pm UTC](https://boards.straightdope.com/t/so-will-anybody-ever-trust-the-polls-again/924714/259 "2020-11-10T14:09:40Z")
So, let me get this straight. Are you guys saying that PredictIt did a better job at predicting Georgia than 538? Because 538 gave Biden a 58% to win, while PredictIt had Trump at about 60 cents. Please explain. ---
**Author:** ![Fotheringay-Phipps](https://avatars.discourse-cdn.com/v4/letter/f/d9b06d/32.png) [@Fotheringay-Phipps](https://boards.straightdope.com/u/Fotheringay-Phipps)\ **Post date:** [November 10, 2020, 3:03pm UTC](https://boards.straightdope.com/t/so-will-anybody-ever-trust-the-polls-again/924714/260 "2020-11-10T15:03:38Z")
> [@Mijin](#): > > Fotheringay-Phipps has said that he “strongly suspects” there is bias towards Trump, “for the reasons given”…where the “reasons given” are just these two, more accurate, predictions. > > These suspicions should be backed up with something, and the fact he can’t do that is why he finally ‘agreed to disagree’ I don’t know what this is about or how closely you’re following the discussion leading up to my remark. My reason for suspecting that PredictIt is skewed towards Trump was given in my first post on this subject, i.e. that PredictIt was also skewed to Trump on election night when that prediction was more wrong than 538. (To that I would add that even now, a full week after the election, when it’s all over but for the shouting and sore-losing, PredictIt is giving Trump a 13% chance of victory. You think that’s an accurate assessment?) In sum, PredictIt is more consistently pro-Trump than it is consistently accurate, which is why I suspect that what drove their higher-than-consensus estimates of a Trump victory is a pro-Trump skew and not an enhanced accuracy. LT responded with some analysis involving mean-square error, but that was logically flawed in context, since it all involved the exact same prediction, just repeated. I pointed this out and he responded by repeating essentially the same argument again, at which point I had nothing further to add. In any event, as to why the betting markets would have a pro-Trump skew, I can think of two possible factors: 1. Emotions. It’s pretty clear that people’s feelings about what they would want to have happen influences what they think will happen. I believe there may be studies of this but you see it in many areas - e.g. die-hard sports fans tend to overestimate their team’s chances, and so on - and in political handicapping as well. As a result, ISTM that there will be a market pricing skew towards even pricing – which overprices the underdog – if two conditions are met. They are 1) the market is dominated by people who have strong feelings on the subject (as opposed to sports situations where most gamblers are not fans of the specific teams playing in a given game, or stock market situations where relatively few people approach it emotionally to begin with), and 2) the people with strong preferences are roughly balanced between the two sides. In such situations, the net influence of the two groups of partisans would be towards even odds, and while this will obviously not produce even odds, it will push the price in that direction, and tend to over-rate the side with lower chances. In the specific case of Trump, he is a guy who inspires particularly strong feelings from a pretty big percentage of the population, so the effect is stronger than normal. 2. Trump’s mystique. Fact is that no one ever thought Trump would win the first time around (538 said he had a [better chance of playing in the NBA Finals](https://fivethirtyeight.com/features/why-donald-trump-isnt-a-real-candidate-in-one-chart/) than of winning the Republican nomination). And both before his win and throughout his tenure as president he survived enough scandal and mayhem to have ended the careers of 50 ordinary politicians. So there was a widespread sense that the ordinary rules of political handicapping don’t apply to Trump, that the polls don’t matter, that nothing else matters, and that Trump could possibly pull it out, and this tended to push up his odds. ---
**Author:** ![iamthewalrus\_3](https://avatars.discourse-cdn.com/v4/letter/i/258eb7/32.png) [@iamthewalrus\_3](https://boards.straightdope.com/u/iamthewalrus_3)\ **Post date:** [November 10, 2020, 4:09pm UTC](https://boards.straightdope.com/t/so-will-anybody-ever-trust-the-polls-again/924714/261 "2020-11-10T16:09:00Z")
I tried to make a bet on the Presidential election on Predictit the day before the election, and got an error about the maximum number of people already participating in the market. It’s not clear to me at what point in time that limit was reached, but I’d be cautious about taking PredictIt too seriously in the future. If they artificially limit market entrants, then at some point the prediction is going to become sclerotic. The reason that Trump has a 13% chance on it today is that it’s not possible to come in and take those people’s money. ---
**Author:** ![Fotheringay-Phipps](https://avatars.discourse-cdn.com/v4/letter/f/d9b06d/32.png) [@Fotheringay-Phipps](https://boards.straightdope.com/u/Fotheringay-Phipps)\ **Post date:** [November 10, 2020, 4:23pm UTC](https://boards.straightdope.com/t/so-will-anybody-ever-trust-the-polls-again/924714/262 "2020-11-10T16:23:43Z")
I don’t understand what you’re saying. That 13% is not some odds being set by someone. That represents an actual bet, with real money on the line on both sides, and the price set by supply and demand. (FWIW, the number on Trump is back up to 16%.) Even if they’ve limited the number of people, it would be the same supposedly accurate people who produced the prior estimates. ---
**Author:** ![iamthewalrus\_3](https://avatars.discourse-cdn.com/v4/letter/i/258eb7/32.png) [@iamthewalrus\_3](https://boards.straightdope.com/u/iamthewalrus_3)\ **Post date:** [November 10, 2020, 4:41pm UTC](https://boards.straightdope.com/t/so-will-anybody-ever-trust-the-polls-again/924714/263 "2020-11-10T16:41:36Z")
I’m saying that the market is not liquid, so we can’t put much faith in the prices it comes up with. In no universe is there currently a 13% chance that Trump wins the Presidency. If PredictIt would let people bet against it, that price would drop dramatically. But they won’t, so it doesn’t. I don’t know when the PredictIt market became illiquid, but I know I couldn’t bet on it the day before the election. ---
**Author:** ![Lance\_Turbo](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/lance_turbo/32/6156_2.png) [@Lance\_Turbo](https://boards.straightdope.com/u/Lance_Turbo)\ **Post date:** [November 10, 2020, 5:19pm UTC](https://boards.straightdope.com/t/so-will-anybody-ever-trust-the-polls-again/924714/264 "2020-11-10T17:19:31Z")
> [@naita](#): > > I say he’s using an inappropriate comparator. I am using a standard comparator for probabilistic forecasts. It’s a comparator that 538 uses to [evaluate its own forecasts](https://projects.fivethirtyeight.com/checking-our-work/president-election-night/). What do you suggest? How should we answer the question, “Which probabilistic forecast was more accurate for the 56 electoral college awarding entities in the 2020 presidential election? 538’s poll driven model, or a model derived from PredictIt using the methodology from Rothschild’s 2009 paper (linked above)?” What comparator should we use? > [@Monocracy](#): > > So, let me get this straight. Are you guys saying that PredictIt did a better job at predicting Georgia than 538? Because 538 gave Biden a 58% to win, while PredictIt had Trump at about 60 cents. No one is saying that. I am saying that when you compute the MSE across all 56 electoral vote awarding entities for the 2020 presidential election for both 538 and PredictIt models, PredictIt outperforms 538. > [@Fotheringay-Phipps](#): > > PredictIt was also skewed to Trump on election night when that prediction was more wrong than 538. PredicIt was more accurate before the election. Things that happened election night and after do not change that fact. [Previous page](https://boards.straightdope.com/t/so-will-anybody-ever-trust-the-polls-again/924714.md?page=12) [Next page](https://boards.straightdope.com/t/so-will-anybody-ever-trust-the-polls-again/924714.md?page=14)