[quote=Nate Silver] What about the polls? Didn’t they show a wider margin for Biden? Yes, they did — Biden led in the final national polls by around 8 points. So we’re probably going to wind up with a polling error of around 3 to 4 points, both nationally and at the state level. (Although that will reflect a combination of states like Georgia, where the polls were spot-on, and others like Wisconsin, where there were big misses.) This is, of course, a subject on which we’ll have more to say in the coming days. For now, it’s safe to say that pollsters will have some questions to answer, especially about how they missed in the same direction (underestimating Trump) in some of the same states two elections in a row.
Note that last sentence. On AVERAGE, Presidential polls are off by 3%. That means half the time, it’s worse than that. Polls are like hammers; useful blunt instruments, but people who expect to be able to use them to perform brain surgery will always be disappointed.
True, and by the time AI can both resolve that ambiguity and predict elections better than us, we’re probably better off handing it the government instead of asking it who will win the election
PredictIt markets do have a tendency of swinging around wildly after the polls close as votes come in. This doesn’t change the fact that PredictIt had lower MSE every single day for months before the polls closed.
It’s crazy to judge PredictIt’s performance by a single data point after the polls closed rather than by evaluating tons of data from before the election.
All that “tons of data” boils down to one thing: PredictIt consistently rated Trump’s chances as higher than 538. Repeat that every day until the election and you get to call it “tons of data” but really it’s only one difference. So the question remains as to whether they consistently rated Trump’s chances higher because they were more accurate or because they just skewed in favor of Trump. I strongly suspect the latter, for reasons given.
Excellent point — surprised I hadn’t heard this yet! Makes sense. It’s a little like how the home team bats in the bottom of the inning, including the ninth. They might have a slight advantage, both strategy and morale-wise, because they KNOW how many runs they must score to win the game.
For each race I calculate the difference between the given forecast probability and the result and then square that difference. I compute the mean of these values across all races.
Why mean square error, by the way? I don’t care how accurate 538 was in May, because I wasn’t making bets based on the information, so I’m not sure why I should care which was more accurate over a large time frame.
What I would really like to see is how PredictIT did against 538, on a given day shortly before the election, for multiple races. We also need to decide how important it is to be on the right side of the answer, as I could have a low mean square error and never get a race correct.
I should note that I’m not disagreeing with anything you’re saying, I just don’t think that your single measure, even over time, tells us much.
PredictIt was more accurate period. They may have been more accurate for the wrong reason, but they were inarguably more accurate at any point in the cycle up to election day.
PredictIt was more Trump leaning than 538. Reality was also more Trump leaning than 538. PredictIt was a more accurate reflection of reality. It’s a simple fact.
So, for example, if I said 70% chance for Biden to win, and 30% chance for Trump to win, my mean square error would be ( (1-.7)**2 + (0-.3)**2 ) / 2 = 0.09?
The average of the square of the error across all 56 electoral vote awarding entities. Every day. This includes every given day shortly before the election. Pick a day. There’s no day in the graph in post #185 where 538 was better than PredictIt. 538’s best day was not as good as PredictIt’s worst day.
If you said 70% chance for Biden to win, and 30% chance for Trump to win and Biden won, your error would be 100% minus 70% which is 0.7. The square of your error would be 0.09. Your mean squared error would be 0.49 because you are only looking at the one race.
If Biden had lost your error would have been 0.7 and its square is 0.49.
Yes, but you haven’t done it for multiple races. That would be the way to determine if Predictit is more accurate than polls in general, or if this is a Trump-specific thing.
Here’s a slope chart of PredictIt probabilities (debiased for favorite/longshot bias) compared to 538 for races where either gave any party a 10% or greater chance to win. There were 18 such races and no surprises outside these 18.
538 is taken from when they froze their forecast early 11/3. PredictIt data was the average of several observations on up to midnight on 11/2. So roughly the same timeframe.
538 was more Biden leaning than PredictIt pretty much across the board.
Here’s that same info in table form with a column for results.
Again, I have no idea what you want me to look at. What is 0.004 and what is 0.100 for SC? I know you keep linking the definition for MSE, but let’s just assume I know what is is (I do). If those are the MSE values for a given day, keep in mind that I can’t actually reverse engineer them to get the source data and I have no idea what day you picked, nor why those 18 states were chosen, nor how well either one did for any race that didn’t involve Trump.
Walk us through the thought process here or something, otherwise I either have to trust your method is ideal or hunt the data down myself. While I have no reason to disbelieve you, every time you post I get more confused at what you are looking at, so I also don’t have any reason to start believing you. Originally you posted what you said were numbers across 56 electoral vote awarding entities over time and now I see 18 entities with some numbers that aren’t well defined. Dude, I wouldn’t be shocked if PredictIt performed better, but help me see it.
Here’s a slope chart of PredictIt probabilities (debiased for favorite/longshot bias) compared to 538 for races where either gave any party a 10% or greater chance to win. There were 18 such races and no surprises outside these 18.
538 is taken from when they froze their forecast early 11/3. PredictIt data was the average of several observations on up to midnight on 11/2. So roughly the same timeframe.
I’m not talking about any races that don’t involve Trump nor making any claims about any such races.
My claim is that PredictIt performed much better that 538 in the presidential election of 2020.
Okay, now I know what 0.100 was and I’ve hunted down where you derived it from 538. I don’t know shit about PredictIt, but can’t find anything about the winner of the voting for president and the probabilities thereof for South Carolina. Not knowing anything about PredictIt, I also know nothing of favorite/longshot bias removal, but based on its name and the fact that you “debiased” it, I’m assuming you are using some formula to counter irrational human behavior. If so, why would you remove that when measuring?
Sorry for being a pain, but while I know what MSE is I don’t spend time on online gambling markets and have no clue how they operate nor where to get the data you are using to show probabilities.
