FiveThirtyEight.com decimated by Disney layoffs

Politics & Elections
21

In the real world of politics all national polls are meaningless noise, or worse, like newspaper headlines of “cures” when the actual articles talk about small advances in a study of rats.

Silver’s advance was in reducing the noise by producing a poll of polls, finding a mean that canceled some of the inherent biases of individual pollsters. The number still meant nothing but it led those interested in the process to look behind the numbers for how polls are conducted and what errors must be overcome to reduce the inadequacies of polling.

One critical point usually gets overlooked. The national data is so worthless that it is given out free. Underlying those gross numbers are the pay-for info. The campaigns pay to find out how middle-age non-college Hispanics with median salaries from $40,000-60,000 respond to the large number of questions surrounding the presidential approval throwaway. That info tells them what areas are worth spending money on, what commercials should run and what they should say, what candidates for the party should officially back, and the other insider stuff that is an open secret that doesn’t make the press releases.

Silver had the misfortune of ripping open the process in the last days of the era of telephone polling, a known whose biases could be mathematically corrected with some precision. The internet destroyed telephone sampling as it was understood. No really good online survey techniques have yet been developed. (Gallup’s 4/27 poll has Biden at 37/59 while Rasmussen has him at 51/49.) The polarization of the public gave them stock answers to political subjects, knowing the answer to the first question predicted answers all the way down. Close elections rely almost entirely on turnout, which cannot be predicted. I don’t know what Silver will do in the future, but polling itself is broken. So is reading comprehension among the public, apparently. He might just go all out on sports, where opinions don’t count and everybody shows up to the games.

22

As a side effect - no Friday puzzle today (or possibly ever again) I’ve tried a few and enjoy reading the solutions even for ones I did not event attempt.

Brian

23

But it never was as high as 95%. I linked you the historical graph. It never was even as high as 90% (though got to 89.7 or something like that.)

And that’s how predictions work. Otherwise, by your definition, almost everything would be a tossup. Remember, 538 had the WORST odds of Hillary winning come November than any other service. They were the ones closest to being right, and if your prediction model is done accurately, you are supposed to/expected to be wrong some of the time, in the long run, otherwise, it should be spitting out 100% chances each time. It’s like sports betting and oddsmaking and all that. On any given gameday, anything can happen, and that’s why we have probabilities and error bars and all that in predictions. They’re predictions. By nature, they are all “tossups” in some way. I can’t, for example, definitively state Purdue will beat Farleigh Dickinson in the NCAA tournament, even though it seems 100% a lock they should. (And they didn’t, for only the second no. 1 seed to lose to a 16 seed in history. Their implied odds were around 95%+ to win. Does that mean the odds were wrong? I say no. They had a crazy game where everything came together.)

Nate Silver was the one who prepared me for the real possibility that Trump could win. Everyone else (it seemed) had Hillary in the bag. So when he won – I wasn’t the least bit surprised, but I was disappointed as hell, to put it lightly.

24

Is this side effect also related to the absence of any coverage of the latest World Chess Championship? I remember Oliver Roeder providing a detailed analysis of past matches (Karjakin challenging Carlsen in 2016, or Caruana challenging Carlsen in 2018), and I was surprised to see no coverage of the current match.

25

The model that I was paying attention to on 538 had it as 2/3rds in her favor. I forget which one but I think it was the one that was purely his, without adding in the outside analysts. The analysts are human, the model is using statistics is just taking economic data, surveys, etc. and doing some math. I trust the model a lot more and I don’t know why he handicaps it.

Statistically, 2/3rds isn’t much better than a coin toss.

26

Yep, and to understand why everyone else was so wrong, pretty much every poll in the 4 or 5 states Trump needed to win showed Hillary in a close lead. Everyone was looking at that going “well, the odds of the polls being wrong in ALL those states is pretty rare, so Hillary’s going to win.” 538’s model was the only one that took into account the idea that if the polls were overestimating Clinton in one state, then it’s likely they’re overestimating her in all those states.

27

Correct. I looked again, per the link someone kindly provided above, and the main page had Trump at about 28% on election night.

It was, in effect, a tossup at that point, and Silver might as well have labeled it as such with a big “T” instead of providing numbers precise to one decimal point.

It’s true I misremembered Hillary’s top number, but I saw when I looked again that it was about 89% on October 16. Sure, there was the Comey affair, but when you have a result that is hyper-sensitive to a few swing states and can change instantly based on the news, then should one ever go that high in the first place?

Probably the honest thing to do–and this is why I got pissed–is say from the start (or at least after one saw clearly the nature of the situation), “The race is a tossup and so labeled, but we lean Hillary based on the numbers. We’ll tell you if and when that assessment changes,” and leave it at that. The trouble is that Silver had a product to sell, and sell it he did–with a lot of confidence and daily chatter.

You can defend his model and say it was better than others; I’m not really disagreeing with that. I am saying that it was presented as something it actually wasn’t.

As an example of what I mean, I examined, as an employee of a company a long time ago, some work a major outside consultancy had performed in order to examine the potential of a new business division. They made two mistakes:

  1. The counted existing business as business that would be newly developed in the division. They had to lower the net present value of the whole thing by 40% once I pointed this out. Oops.
  2. More insidiously, however, the success of the new division was about 90% due to the success of a single product–a product that the company had struggled to find success with in the first place. Oops again. They had performed no sensitivity analysis with their data.

I think Silver’s error was like #2 above. He put big fat percentages on the main page, yet those were too much influenced by too few factors (swing state polls) to be of much value.

In 2020, I ignored his site and just listened to general news about polls, which seemed favorable to Biden, and I hoped they were right. Whereas I had slavishly followed 538 on the daily in 2016. You may disagree with some of what I wrote above, but would anyone here say that that was a bad decision? I.e., would anyone here contend that Nate Silver’s website (whatever it will be called) is worth special attention in the future?

(By the way, the consensus after 2016 seemed to be that the polls had significantly underestimated support for Trump where it counted most. In 2020, the Trumpists were arguing that the same thing was happening and that Trump would win despite the bad poll numbers–but they were wrong. In 2016, although Silver’s model did better than others as people here say, he was still working with bad numbers, which were enough to render his model more or less useless. So part of my takeaway from 2016 was that we never really know how good the polling numbers are, so ultra-precise models like Silver are more or less worthless whenever the race is anywhere close.)

28

Ideally, he would learn from past mistakes and figure out how to incorporate an amount of “stickiness” into future models the further out from election day it is.

But, if Comey had made his announcement just a few weeks later, it could have cemented the result. Those spikes can matter - depending on how far they are from actual voting day.

29

You’re being disingenuous and cherry picking the maximum prediction of the model at one point in time (and well before election day), which also had the election at a toss-up in late July and nearly so through the latter half of September. The model constantly predicted a statistical overlap of more than 270 Electoral College votes for either candidate, and overlapping in majority popular vote for the entire election. Prediction models are only ‘right’ or ‘wrong’ in their utility to estimate trends and propensity to some useful level of accuracy, and even a cursory look at the predictions of the FiveThirtyEight 2016 Presidential Election model clearly indicated that there was a not-insigificant chance that Trump would win the Electoral College vote.

Stranger

30

Yep. One of the hardest things to deal with in gambling or prediction is that you can get a prediction wrong even if the model,was completely correct, or get a prediction right even if the basis for the prediction was hot garbage. It’s very tempting to throw out a model that gets an answer wrong, or elevate one that gets the answer right. When the answers are in part statistically based, that can get you into trouble.

This is the essense of every gambler’s fallacy - in a statistical world, sometimes bad ideas leed to good results, and good ideas fail. Nate Silver said Trump had about a 25% chance of winning, and sure enough, Trump won. Those facts don’t tell us anything about the validity of Silver’s model, good or bad. But he does have a track record of being better than most at the prediction game.

31

75% does NOT mean ‘tossup’. A Tossup happens when no one has a statistical advantage, and either outcome is statistically as likely. Anyone calling 75-25 a ‘tossup’ is doing it wrong.

Also, if the 25% chance comes through, that does not make it a tossup either, nor does it invalidate the 75/25 prediction. You’d have to run many elections before you could see if the model was correct. And of course you can’t do that, because the world changes between elections, invalidating prediction models.

Therefore, you should take all election predictions with a grain of salt. Most of them are garbage no better than simply observing who looks more popular at any given time. Trafalger got all kinds of praise for being so close in 2016. Then they were completely wrong next time around.

If everyone throws darts at a dartboard, someone has to be closest to center on each round. That doesn’t mean they were any better than the others. If they do it 10 times in a row, then maybe we can use that data to draw some conclusions. In political prediction, we can’t do that. It’s always a single round.

32

Isn’t calling someone “disingenuous” a statement about their intentions? I ain’t trying to fool anyone here. I’m trying to explain my decision never to follow 538 again, to explain my reasoning for doing so (which requires me to go back in time to my mind in 2016), and to reassert (to myself and to you all) that I think I made the right decision.

You can argue that the model was correct, statistically speaking, and I’m saying that, even if it was correct, it wasn’t useful, even though Silver wanted to portray it as useful (otherwise he wouldn’t have had a product to sell).

My current mental algorithm is this: either the polls are unambiguous (e.g., Clinton was going to win in 1996; Bob Dole had no chance, and no one ever thought he did) or something less than that. If anything less than that, assume it’s a tossup and fight like hell for the Democrat–or at least get out and vote for them. Fancy models mean nothing in such a case.

33

No doubt. But keep in mind that Silver’s model wasn’t modeling something like tosses of a fair die or something mathematically certain like that. How to turn disparate poll numbers into a single win/lose percentage will always be a matter of human judgment and opinion. I, the reader, must in turn make my own judgment as to what that 75% actually means in the real world. And I decided that it didn’t have any meaning other than perhaps the person with 75% had a slight to moderate advantage.

Correct. So the person who makes such a model can never be “wrong,” since they can always say, “Well we said there was a 1 in 4 chance of so-and-so winning!”

Yep, that’s what I’ve been saying the whole time here. Silver’s model should be taken with a handful of salt and isn’t worth paying attention to.

34

Well, it wasn’t useful for your expectation, which is apparently to give an absolute prediction of a binary choice, which is something no statistical model will do by definition. It was clearly very useful in processing polling data and historical predictions to show that Hillary Clinton had an often narrow lead and there was a significant likelihood that Donald Trump could be elected.

Stranger

35

Yes, had I been an idiot, sure, then that’s how I would have thought.

I don’t agree that it was particularly useful. More useful would have simply to have posted a paragraph on the main page saying, “Polls are so tight in swing states that it would be imprudent to try to make any concrete prediction about the election, but we see Hillary as having a slightly better chance of winning.”

36

If you wouldn’t play Russian roulette then we’re effectively agreed that, for matters of grave importance, 75% is close enough to a tossup as to be an issue of semantics. It’s so close to being very very real that, again, for matters of life and death, national security, etc. it’s well under the threshold of any sane safety margin. You don’t want to be in that zone.

Mathematically, obviously it’s not, but even if Trump had a 99.9% chance of losing and, nevertheless, won then - in mathematical land - that’s completely reasonable. But in human psychology, the math was wrong and a lie even though the math said there was still a completely reasonable chance that the guy could win.

In terms of how to report the math, for non-math humans to brain reasonably, 75% is effectively a tossup.

37

Thank you, Sage_Rat!

38

You mean like this:

and this:

and this?:

FiveThirtyEight is predicated on providing quantitative analysis. If what you are looking for is just a condensed qualitative summary based upon someone’s ‘gut feel’ then it clearly isn’t the site for you.

Stranger

39

Nope. That’s just adjunct information to support the big number at the top of the page.

Right, if I were an idiot, then I would want someone’s “gut feel” and would go with that.

The site is for laypeople, perhaps more sophisticated laypeople who have, as I have, studied statistics.

Note that I have never once said that Silver’s top number should have been X instead of what it actually was. I am saying that the site presented that information as having meaning and utility that it didn’t actually have.

So we are arguing at cross purposes. You are, it seems, defending the model and its numerical outputs. I am not saying the model was bad per se as a model. I am saying that the model ultimately didn’t offer meaningful information, and it presented essentially meaningless information as being meaningful.

40

Silver is actually the person who tried to teach people about statistics and polling. If you think his model predicting a 75% chance of a win meant anything other than 3 times in four similar occasions one person would win, and one time in four the other person would win, then he sadly failed with you.

Perhaps you, with your sophisticated understanding from having studied statistics, went into the site with preconceptions that precluded you from reading and understanding what Silver was actually doing. People can fool themselves that way. I’ve studied statistics myself but I realized that his approach was itself the sophisticated one, and had great amounts of meaning and utility, both for laypeople - the great majority, remember - and our little band of learned sophisticates. Sorry he failed with you; you missed out on something good.