By the way, at a glance it appears that they bin continuous data, such as margins, and then price those bins, but for categorical data, such as win/loss, they simply assign a price. It also appears that the value of all bins is at or less than a dollar. So, I assume that you’re saying the price for a Trump win in SC on PredictIt was 99.6 cents, but I can’t find that on their site.
I built a robot to use the PredictIt API to grab a snapshot of each of the 56 electoral vote awarding entities’ markets several times a day starting back in April. Each snapshot has a price for the Democrat and the Republican to win that contest (e.g. $0.29 D and $0.73 R). These will add up to more than a dollar due to the overround.
I normalize these amounts so that they sum to one. Then I debias for favorite/longshot bias. Favorite/longshot bias is a well known, and easily correctable, weakness of the predictive value of betting markets.
I use the method described by David Rothschild in his Public Opinion Quarterly paper Forecasting Elections: Comparing Prediction Markets, Polls, and Their Biases. The debias function is defined p’ = Φ(1.64 * Φ^(-1)(p)) where Φ is the CDF of the standard normal. The number 1.64 is a number other researchers determined which is cited by Rothschild.
So there is some slight transformation of the raw prices on PredcitIt to get to the probabilities for the PredictIt derived forecast model, just as 538 transforms raw poll data to get to their model.
Let me know if you have any other questions.
Here’s a link to Wikipedia’s page on mean squared error: MSE. 
Okay, then it sounds like my assumptions above were pretty accurate. I don’t see any “market” (I think that is the terminology that they use for this) that shows value for the state of South Carolina presidential outcome.
MSE.
South Carolina market: Which party will win South Carolina in the 2020 presidential election?
It’s closed now. On election day it was somewhere around $0.94 Trump, $0.07 Biden.
You can’t get a ton of historical data off of there. That’s why I built a script to use their API and build my own database.
It’s a ~58k row 4.6 MB csv that I haven’t put online anywhere, but I suppose I could if anyone really wanted it.
Well, you almost had me fully on board, but just confused me again. How are you deriving 0.004 probability from $0.07?
Edited to add (I forgot): MSE 
From a previous post:
I normalize these amounts so that they sum to one. Then I debias for favorite/longshot bias. Favorite/longshot bias is a well known, and easily correctable, weakness of the predictive value of betting markets.
I use the method described by David Rothschild in his Public Opinion Quarterly paper Forecasting Elections: Comparing Prediction Markets, Polls, and Their Biases. The debias function is defined p’ = Φ(1.64 * Φ^(-1)(p)) where Φ is the CDF of the standard normal. The number 1.64 is a number other researchers determined which is cited by Rothschild.
Also, that $0.07 is the best sell offer. I use the average of the best sell offer and the best buy offer as the current price. So it will start off a little lower than that. Additionally I normalize both prices so that they sum to one so that could move things a bit before debiasing. Finally everything is smoothed to some degree so it may not match any particular snapshot in time.
Got it, so the debias algorithm is what is taking it from $0.07 to 0.004. I’d be curious how the scores pan out without that extra step, but I don’t think I care enough to push you to do it, but you do have to acknowledge that comparing 0.07 to 0.1 is a lot closer than 0.004 to 0.1. Here, I’ll state it for the record:
If you debias PredictIt data, to account for people under/over valuing a given “market” buy/sell, it outperforms 538 for the presidential election in 2020.
Do you have typos here? My example result was Biden at 70% gives a mean square error of 0.09.
I do.
The square of your error would be 0.09. Your mean squared error would be 0.49 because you are only looking at the one race.
Should be.
The square of your error would be 0.09. Your mean squared error would be 0.09 because you are only looking at the one race.
Your example is wrong because it gives the same result regardless of the actual winner.
Your comments were just to repeatedly say that the prediction markets leaned more towards trump.
But while this is comparatively true, they still were saying Biden was the favorite, just with less margin than the polls were suggesting. i.e. they made a better prediction.
I’ll let @Fotheringay_Phipps speak for themselves, but what I took from that wasn’t that he/she was questioning the performance of this particular race, but whether the performance was actually the result of the model being “better” or “lucky”. It’s a valid question and I don’t believe it’s been answered yet.
I will say that I’m not questioning the numbers provided by @Lance_Turbo, and think quite highly of his analytical abilities, as can be seen in that silly excess deaths thread that he and I participated in, but I have yet to see the necessary data to lead me to believe that betting markets consistently outperform the polling scoring and aggregation method. I don’t even think I’d be shocked, as I’d assume the efficient market hypothesis would come into play in the betting markets, at least to an extent.
For the record, my claim is that a probabilistic forecast for electoral vote awarding entities in the 2020 election derived from PredictIt data outperforms 538’s probabilistic forecast.
However, I used the exact method described in the paper I linked in post #226, and that paper showed that an Intrade derived model beat 538 in 2008, although not as decisively (see figure 4). There’s also many references in that paper that cover this same topic with similar results.
Note: Intrade does not exist in 2020 and PredictIt did not exist in 2008.
So far no one has provided any data that shows that 538 or any poll driven model has ever outperformed prediction markets.
That’s a fair point and to be honest I’m still debating whether I care enough to pursue it. Right now I’m leaning towards not bothering, but perhaps that’s simply because while I defend Nate Silver and his methods, it’s a lot harder to defend pollsters, other than to say it’s hard to do well, even for honest ones.
If Trump had won, my 70% Biden and 30% Trump probabilities would give a score of ( (0-.7)**2 + (1-.3)**2 ) / 2 = 0.49 by my reckoning. What would you get?
You know what, I misread your original expression.
You’re calculating the error for the Biden prediction and squaring it, calculating the error of the Trump prediction and squaring it, then averaging those two. That’s perfectly fine, but the square of the error of Biden prediction and the square of the error of the Trump prediction are always the same so you’re doing a little extra work.
We have probabilities p and q = 1 - p and a result X in {0, 1} with Y = 1 - X.
Then (X - p)**2 = [(X - p)**2 + (Y - q)**2]/2 because (X - p)**2 = (Y - q)**2.
That’s the squared error for one race. To get mean squared error just average that across how ever many races you have, in this case 56 (or 18 in the smaller example).
For more information as to why this is proper way to evaluate probabilistic forecasts see: Brier Score.
Okay. I understand what you’re doing. I’d do a square-root of the mean of squares of errors, just to keep the units in probability, but that’s a matter of taste.
I think it’s better to use my more general formula to account for cases with more than two possible results. Are you including any long-shot presidential bets?
No.
I normalize each race so that P(D) + P(R) = 1. If there were a viable third party candidate things would get a lot tougher, but this year was easy in that regard.
I’m curious how much of a difference it’d make in the scores of all the long-shots were included.
Links to data sets…
PredictIt Raw: https://drive.google.com/file/d/1N08KwTI_NpCHdAdm4kItPdleeR956imA/view?usp=sharing
PredictIt Normalized, Smoothed, and Debiased: PredictIt.csv - Google Drive
538: https://projects.fivethirtyeight.com/2020-general-data/presidential_state_toplines_2020.csv
I disagree. It’s like saying that the last two coin flips were heads so maybe it’s a biased coin?
Because firstly, I didn’t say anything about only considering these two elections. The prediction markets have been producing good results (not always, but compared to other methods like polls) for many years now. There’s plenty of data to help us ascertain how often prediction markets correctly predict outcomes.
Because secondly, if we were insisting on putting on blinkers and looking at just the Trump election(s), the answer to that question would be basically unknowable. How could we rule out a net bias for a system that fundamentally is the aggregation of individual bias? We can’t.