Election predictions: InTrade vs. 538

[Ravenman]

The New York Times blog 538 keeps a running tally of a carefully designed model it uses to predict the popular and electoral vote based on recent polls. As of this afternoon, the model states that Obama has a 63% chance of winning re-election.

InTrade is a political predictions market in which people invest real money in futures of whether a particular event will happen or not. The thought is that knowledgeable people with proper incentives end up being pretty good predictors of discrete events. Right now, InTrade is predicting about a 53% percent chance of an Obama re-election.

From where I sit, the difference between a 53% chance of something happening and a 63% chance of it happening are pretty substantial. So, which prediction do you think is more accurate, and why?

[Simplicio]

Intrade seems to basically just track conventional wisdom, which in turn (somewhat illogically) seems to track polls. So if I had to bet, I'd bet InTrade is underestimating the chance of an Obama victory by a few points.

[Leaper]

I'd agree. 538's predictions seem to come from polls. InTrade's prediction seem to come from how people feel about the polls. One has one less layer of subjective thinking than the other.

[furt]

It comes down to whether or not you think polls are an accurate gauge of what people really think and how they will really vote (or more precisely, what people really will think and how they really will vote in November).

Silver's model is as objective as he can make it, which is pretty damn good; but it also precludes him from making judgements about "people are saying X, but they don't really mean it." That kind of perceptive judgement is real, and some people base their whole careers on it (though usually at the individual, not societal, level).

Of course, we don't know if intrade bettors are more or less perceptive than the average person. If 90% of them were ad execs, psychologists and car salesmen, I'd give it a lot of weight. As it is ... maybe, maybe not.

[Measure for Measure]

538's economic variables don't weigh the odds of a downturn sufficiently. Economic forecasters tend to provide median forecasts, which don't reflect events like, say, Greece leaving the Euro. More prosaically, forecasters aren't especially talented at predicting recession. I opine that InTrade's take is closer to the mark. I suspect that during more typical economic conditions, that the two sources would agree more.

The other unmeasured factor is Romney's Super-Pac money. I discount that, or rather I think it's offset by the personal unattractiveness of the presumptive GOP nominee.

Finally, InTrade incorporates 538's forecasts. The opposite is not the case.

ETA:

Originally Posted by Ravenman

From where I sit, the difference between a 53% chance of something happening and a 63% chance of it happening are pretty substantial.

The difference is worthy of a thread, but offhand if one person thinks Obama's odds are 50-50 and another thinks they are 60-40, then I really don't believe they disagree too much. If they are loss-averse a 55-45 bet would not be attractive to either.

[brickbacon]

Does InTrade actually take money? Because if they do, the odds are just as much about providing a market with a roughly equal number of people on each side, as they are about predicting who wins. That said, you would expect their money odds to skew more towards the underdog since that will even out the betting.

[Chronos]

There have been times in the past when Intrade's odds on political events have fluctuated wildly due to the actions of one individual bettor, or a small number of them. The concept of crowdsourcing information, which a betting site like Intrade depends crucially upon, breaks down when a single outlier can swing the results so far. And you've also got to consider the demographics of the crowd that Intrade is averaging over: There's a natural tendency for people to overestimate "their side", and Intrade probably attracts a more conservative audience than the country as a whole.

[Chronos]

brickbacon, the way that InTrade is set up is as a forum for bets between private individuals (with the site taking a cut, of course), so they inherently have the same number of bettors on each side. The prices that those individuals agree on can then be straightforwardly converted into probabilities, if you assume that the bettors are rational and that they collectively have access to the true probabilities.

[Measure for Measure]

Originally Posted by Chronos

brickbacon, the way that InTrade is set up is as a forum for bets between private individuals (with the site taking a cut, of course)...

The site no longer deducts transaction fees. Instead they charge members $5 per month.

[Bricker]

What are the track records of each?

[ultrafilter]

As I understand it, the theory that election-oriented prediction markets are based makes two pretty questionable assumptions:

1. Everyone who participates is betting sincerely (i.e., in line with their actual beliefs).
2. The participants are representative of the electorate as a whole.

The latter is definitely an issue here, and the former is somewhat questionable given the money that's up for grabs. Nate wrote a short piece about irregularities at Intrade during the 2008 election which is worth taking a look at.

Originally Posted by Bricker

What are the track records of each?

If I claim that an event has a 90% chance of happening and it doesn't, what's my track record?

[Measure for Measure]

Originally Posted by ultrafilter

As I understand it, the theory that election-oriented prediction markets are based makes two pretty questionable assumptions:

1. Everyone who participates is betting sincerely (i.e., in line with their actual beliefs).
2. The participants are representative of the electorate as a whole.

But the participants aren't expressing their preference in the election market. They don't have to be representative, they only have to be smart, at least collectively. Of course if there is partisan bias in their judgments, then the composition of the population will matter, absent "Smart money" that can take advantage of such bias.

Bias probably exists: according to Intrade Ron Paul has a 1.9% chance of being elected President. Even given speculation of a third column of RP delegates, that seems high.
---

Intrade is subject to wild swings. But those typically self-correct as profit takers push prices back to a more reasonable zone.
----
Here's a table comparing 538 and intrade:

	538	intrade		diff
				
Obama	0.627	0.538		0.0890
Romney	0.373	0.424	       -0.0510
				
total	1.0000	0.9620

Intrade thinks that the odds of neither Romney nor Obama being elected is currently 3.8%. The punters seem to give a hefty weighting to wild-card events.

If I claim that an event has a 90% chance of happening and it doesn't, what's my track record?

I understand that there is a literature devoted to scoring algorithms. Presumably a substantial sample of predictions is required.

[ultrafilter]

Originally Posted by Measure for Measure

I understand that there is a literature devoted to scoring algorithms. Presumably a substantial sample of predictions is required.

There is. My point is that the everyday notion of a track record is hard to apply when you're making probabilistic forecasts.

[Lantern]

Originally Posted by Measure for Measure

538's economic variables don't weigh the odds of a downturn sufficiently. Economic forecasters tend to provide median forecasts, which don't reflect events like, say, Greece leaving the Euro. More prosaically, forecasters aren't especially talented at predicting recession. I opine that InTrade's take is closer to the mark. I suspect that during more typical economic conditions, that the two sources would agree more.

Yes, 538's methodology is fairly mechanical. This is not necessarily a bad thing but it does mean that it's not really good incorporating something like a belief that there is a 30% probability of a complete Euro meltdown which triggers a financial crisis in the US. If you look at Intrade's history, Obama was at around 60% until the last bad jobs report. This lends support to the idea that Intrade investors have a more pessimistic view of the economy and how it impacts Obama.

Having said that my personal view tends towards 538's; I think Obama has a 60-65% chance of winning. I think the Greek election and the new bailouts have seriously reduced the probability of a Euro catastrophe in the next six months. It may be a case of kicking the can down the road but that's all Obama needs from an electoral perspective. I think a mediocre economy is already priced into his poll numbers so it won't really hurt him more. A silver lining for him is that a mediocre global economy will lead to lowish oil and gas prices something which is already happening just in time for the US summer driving season.

The other big variable: IMO Obama is fundamentally a better campaigner than Romney something that will make a big impact when the actual campaign heats up in the fall. Romney made an awful lot of unforced errors during the primaries against weak opponents. I expect similar or even bigger errors in the heat of a close general election against a much better opponent.

I think money will largely be a wash but organization could be another moderate advantage for Obama. I have no doubt that Romney will build a fine organization but Obama's team have done it all before and will probably have a much richer data set from 2008.

Finally I expect Obama to continue using his agenda-setting powers as President to box Romney into awkward positions just like he did with immigration.

[Ravenman]

Originally Posted by Bricker

What are the track records of each?

From what I can gather, 538 predicted that Obama would receive 349 EVs in the 2008 election, and InTrade predicted 364. The actual was 365. 538 wasn't around in 2004, but I have read that InTrade was correct on its predictions for every state. 538 was correct on every Senate race in 2008, but I'm having trouble finding other data on Senate races.

InTrade is not infallible, of course. It seems that InTrade predicted Obama would win New Hampshire in the 2008 primary, it appears by a significant margin, but I can't find the exact numbers. Here and here are some Beltway Bandit types discussing the pros and cons of InTrade, although not in great depth.

I'll lay it out there: I think the InTrade model just isn't good. Although I follow politics closely, I believe I'm more knowledgeable about world events, and the predictions made on InTrade about the probabilities of various events (e.g., bombing Iran) are almost ludicrous in their lack of connection to reality. I'm exaggerating here, but it as if Obama mentions Iran in a speech and the market suddenly spikes with John McLaughlin-like metaphysical certainty that Iran is going to get bombed in a few days... until it doesn't happen. Basically, I think the main customers of InTrade don't know very much.

However, in this case, at this point in the race, I think InTrade is a better reflection of where the race stands. Not that political prognosticators are infallable either, but Charlie Cook, the highly regarded analyst, recently called the race between Obama and Romney a "coin flip." If I were to put a number on it, I feel like 55% chance of Obama re-election is about right, right now.

[BrainGlutton]

Originally Posted by Ravenman

I'm exaggerating here, but it as if Obama mentions Iran in a speech and the market suddenly spikes with John McLaughlin-like metaphysical certainty that Iran is going to get bombed in a few days... until it doesn't happen. Basically, I think the main customers of InTrade don't know very much.

The stock market reacts wildly to any rumor or poll or election or event that might even conceivably have some effect, good or bad, on any business interests. Investors' anxiety about/hopes for their money seems to be an enormous psychological force-multiplier. It creates a situation where a given stock's market price at the moment sometimes bears no conceivable relationship to how well or badly the corporation is doing at the moment.

[Lantern]

Originally Posted by Ravenman

From what I can gather, 538 predicted that Obama would receive 349 EVs in the 2008 election, and InTrade predicted 364. The actual was 365

This is less impressive than it appears. The difference is because of North Carolina which Obama won by 0.3% which is basically a coin flip.

[Lantern]

Originally Posted by ultrafilter

There is. My point is that the everyday notion of a track record is hard to apply when you're making probabilistic forecasts.

Not really. There are fairly standard methods of measuring goodness of fit in binary models. For example one can calculate the correlation coefficient between predicted probabilities and outcomes. E.g. you create a variable with value 1 if the event happens and 0 if it doesn't and then calculate the correlation with the predictions you make. This is more fine-grained than just looking at correct predictions because for example you score higher if you correctly predict something with probability 90% than with 60%. Of course as mentioned by MoM you need a reasonable sample for such goodness of fit measures to make any sense.

I don't think there is a large enough sample of common predictions by Intrade and 538 but in a few election cycles it should be possible to calculate which one performs better.

[gamerunknown]

Originally Posted by Measure for Measure

The punters seem to give a hefty weighting to wild-card events.

Perhaps it's a risk measure? Given the odds, such an event could shut them down, while 538 would probably just readjust its models. It'd be in their interests to under-estimate any third party populists opposed to gambling in any scenario, since they'd lose out whether they were elected or not.

[Ravenman]

Not directly on topic, but InTrade blew it on the health care mandate. For the past month, the bettors at InTrade believed that there was greater than a 70% chance that the mandate would be unconstitutional.

Oops.

[Chronos]

You can't really call "oops" on a single prediction, though. I mean, even if they got the true a priori probability right, they'd still have failed more than 1 time in 4. Seeing something that has a 1 in 4 chance but still happens is hardly earthshaking.

[Measure for Measure]

...and it was a 5-4 decision with Roberts changing his vote at the last minute:

It is impossible for a lawyer to read even the first few pages of the dissent without coming away with the impression that this is a majority opinion that at the last moment lost its fifth vote. Its structure and tone are those of a winning coalition, not that of the losing side in the most controversial Supreme Court case in many years. But when we get to Page 13, far more conclusive evidence appears: No less than 15 times in the space of the next few pages, the dissent refers to Ruth Bader Ginsburg’s concurring opinion as “Justice Ginsburg’s dissent.”

http://www.salon.com/2012/06/28/did_...ote/singleton/

[Fear Itself]

Originally Posted by Ravenman

Not directly on topic, but InTrade blew it on the health care mandate. For the past month, the bettors at InTrade believed that there was greater than a 70% chance that the mandate would be unconstitutional.

Oops.

How is that inconsistent with the decision of the court? Intrade showed a 30% chance it would be upheld. It was upheld.

Intrade on Obama has gone up for two days since the SCOTUS decision; Romney has gone down for two days.

[BrainGlutton]

Originally Posted by Fear Itself

How is that inconsistent with the decision of the court? Intrade showed a 30% chance it would be upheld. It was upheld.

Intrade on Obama has gone up for two days since the SCOTUS decision; Romney has gone down for two days.

Nonsense. Mormons don't go down.

[septimus]

Originally Posted by Chronos

There have been times in the past when Intrade's odds on political events have fluctuated wildly due to the actions of one individual bettor, ...

The use of a market like that for prediction requires that there be large money action. I don't think Intrade's action is large enough to qualify. I've lost the URL for the site that synopsized all sportsbook positions, but AFAIK the money volume figures would be unavailable there anyway.

Originally Posted by Chronos

Seeing something that has a 1 in 4 chance but still happens is hardly earthshaking.

Indeed one can suspect the predictions are poor unless about 25% of the 1-in-4 longshots succeed.

[ultrafilter]

The standard evaluation for a probabilistic forecaster is what's known as being calibrated, which means exactly what septimus said: out of all the events that they claim has probability p of happening, the proportion that happens is about p. An uncalibrated forecaster is definitely bad, but a calibrated forecaster can actually also be pretty awful. For instance, if it rains 25% of the time, a weather forecaster can predict a 25% chance of rain every day and be calibrated but of seriously questionable value. There's a nice article here that spells this out a bit more, explains some of the other approaches, and has links for those who'd like to a bit of follow-up reading.

[Measure for Measure]

Originally Posted by septimus

The use of a market like that for prediction requires that there be large money action. I don't think Intrade's action is large enough to qualify. I've lost the URL for the site that synopsized all sportsbook positions, but AFAIK the money volume figures would be unavailable there anyway.

That's a big issue. I argue that those who think they are smarter than intrade can make big money betting against it. Since they don't, I assume that their opinions are at least partly hot air.

This is a dubious argument.

Consider the odds of an overt airstrike on Iran. Total volume has been around 10 on most days, though it peaked at 150 on June 28th. The price was around $2.20 per share, implying 22% odds. So maybe $330 of bets were made on that day for a Dec 2012 payoff of $330 or $1000 depending upon what happens. That's a thin market. http://www.intrade.com/v4/markets/co...tractId=750356 Similarly the bid-ask ranges from 17.8% (to buy 2 shares) to 21.8% (to sell 11 shares).

Now consider the odds of an Obama victory. Daily volume averaged 2164 shares last month. Cool. Average closing prices were 5.35 (or 53.5%). That amounts to less than $12,000 per day. By way of contrast the small cap company Yelp had an average daily volume of 660,000 shares, priced at $22. Multiply it out and get $14,520,000 per day -- over a thousand times higher. This for a company with negative profits, revenue of $83 million and maybe 150 employees. So much for speculation about the leader of the free world.

[septimus]

Originally Posted by ultrafilter

The standard evaluation for a probabilistic forecaster is what's known as being calibrated ... There's a nice article here that spells this out a bit more, explains some of the other approaches, and has links for those who'd like to a bit of follow-up reading.

Interesting links. I learned a little about the problem (though less about the solutions ) when I was involved with data compression algorithms. (Probability estimation is often at the heart of data compression.)

There are "paradoxes." For example, the proper "average" of two 90% predictions might be more than 90% if the two predictors are uncorrelated!

[ultrafilter]

Originally Posted by septimus

Interesting links. I learned a little about the problem (though less about the solutions ) when I was involved with data compression algorithms. (Probability estimation is often at the heart of data compression.)

There's actually a very strong sense in which probability estimation, data compression and gambling are exactly the same problem. I would love to find an article that explains that in a way that a non-mathematician can understand, but I'm not holding my breath.

[Bob Ducca]

An update on the 538 numbers, Nate Silver's model has Obama at a 67.8% chance of winning the electoral vote, the highest his chances have been since the model began.

[Fear Itself]

The Intrade on Obama has been a near straight line increase since the SCOTUS decision on Obamacare.

[c007224]

Success, what everyone here fails to mention is 538 is done by nate silver a statistical genius who precicted all but one state in 2008, and every single senate race.....ill go with the statistical genius vs. this market crap

[Napier]

Intrade is a futures marketplace, not a prediction service. Claiming it predicts the future is misapplying what it was set up for.

Traders could hedge their bets elsewhere by investing at Intrade. Suppose you had a financial interest in seeing "Obamacare" overturned, but you wanted to trade away some of your leverage to have greater security. You could invest in Obama winning the election, at Intrade. Now your future earnings are more predictable (though you stand to win or lose less). This is, after all, the nominal reason that other commodity futures markets were established in the first place. Others treating your investment as an informed prediction on your part is obviously reading too much into it.

It does sound as though Intrade does tend to predict the future, but this is inadvertent. The degree to which this tendency persists could well change on the basic issues in the election per se -- that is, Intrade's accuracy could be confounded with the subject matter itself. Silverman, on the other hand, is famous for trying specifically to predict the future, and doing very well at it.

[ShibbOleth]

They evolve over time and will approach each other as the election nears.

[zamboniracer]

Both 538 and Intrade could both be right, of course.

As I understand them, 538 takes the polls per state and applies them to project the Electoral College vote, while Intrade is more is more like a nationwide poll that doesn't account for the Electoral College.

[Fear Itself]

Originally Posted by zamboniracer

As I understand them, 538 takes the polls per state and applies them to project the Electoral College vote, while Intrade is more is more like a nationwide poll that doesn't account for the Electoral College.

Intrade takes bets on the outcome of the Electoral College.

[adaher]

538 also takes into account economic factors. They just slightly reduced Obama's chance of reelection based on a manufacturing report.