# The weirdness of polls... What's up? **URL:** \ **Category:** Politics & Elections\ **Created:** [October 17, 2012, 5:39pm UTC](https://boards.straightdope.com/t/the-weirdness-of-polls-whats-up/638033 "2012-10-17T17:39:08Z")\ **Posts on this page:** 20\ **Page:** 5
**Author:** ![Richard\_Parker](https://avatars.discourse-cdn.com/v4/letter/r/35a633/32.png) [@Richard\_Parker](https://boards.straightdope.com/u/Richard_Parker)\ **Post date:** [October 19, 2012, 7:18pm UTC](https://boards.straightdope.com/t/the-weirdness-of-polls-whats-up/638033/81 "2012-10-19T19:18:44Z")
You might think of an election as involving three variables if you want to analyze things the way OMG has: the size of each group (i.e., overall number of Republicans), the turnout of each group (i.e., how many Republicans show up to vote), and the split among each group (i.e., how many Republicans are voting for Romney). You cannot ignore any of these variables because a candidate could be getting whalloped by two of the measures but win easily because of the third. That’s why it’s perfectly possible for Obama to win even though he wins a lower percentage of Republicans and independents than last time and holds his level steady. All it requires is for there to be some more Democrats, or at least more showing up at the polls. That’s why the key premise of OMG’s is about the enthusiasm gap (which manifests as both a likely voter gap and a self-ID gap). This is the hardest stuff to measure, not only because it involves more slippery questioning but also because it moves most quickly and fluidly based on the campaign (whereas deciding whom to vote for does not move that much). The whole “oversampling” nonsense was largely a question of enthusiasm surges reflecting themselves in differential self-ID. You’ll note that after the first debate when Romney got a surge of enthusiasm, the so-called oversampling seemed to fade way. The bottom line is that there is one reliable way to assess all of these variables at once: ask people how they identify, whether they are likely to vote, and whom they are likely to vote for. You still have to adjust the samples to meet stable demographic targets (which is the sophisticated version of the oversampling argument), but the possibilities for systemic error here are lower because demographic make-up of the electorate is much, much more stable than self-ID and enthusiasm. ---
**Author:** ![Lantern](https://avatars.discourse-cdn.com/v4/letter/l/d26b3c/32.png) [@Lantern](https://boards.straightdope.com/u/Lantern)\ **Post date:** [October 19, 2012, 7:27pm UTC](https://boards.straightdope.com/t/the-weirdness-of-polls-whats-up/638033/82 "2012-10-19T19:27:20Z")
> [@Leaper](#): > > Okay, here’s what I’m not getting (and this is probably pre-lunch famish syndrome, so please bear with my stupid questions): what, specifically, does any of that have to do with **OMG’s** assertions (in post 58 and 61) that Romney is winning independents and not lost any of McCain’s '08 support, that Obama has lost a lot more of his '08 support, and that given that, the topline can’t possibly be right because that means Romney should be way ahead? Well for one thing Obama beat McCain 53-46 so it’s quite possible for him to lose a whole bunch of support from 2008 and still beat Romney by half a point. In fact quickly doing the math in my head the numbers in that post are pretty much what you would expect. ---
**Author:** ![Fotheringay-Phipps](https://avatars.discourse-cdn.com/v4/letter/f/d9b06d/32.png) [@Fotheringay-Phipps](https://boards.straightdope.com/u/Fotheringay-Phipps)\ **Post date:** [October 19, 2012, 7:47pm UTC](https://boards.straightdope.com/t/the-weirdness-of-polls-whats-up/638033/83 "2012-10-19T19:47:53Z")
> [@Richard\_Parker](#): > > Huh? His claim is “Gallup Performs Poorly When Out of Consensus,” and that “The context is that its _most recent results_ differ substantially from the dozens of other state and national polls about the campaign. It’s much more likely that Gallup is wrong and everyone else is right than the other way around.” (emphasis added.) > > You seemed to be disputing that, but now you concede that they’re an outlier. So your complaint is that he’s wrong to have identified previous times when they were also an outlier? What I object to is the dichotomy of Gallup vs. “everyone else”. Gallup is part of everyone else, and the average of all polls would include Gallup. If you set up Gallup vs. everyone else, of course you would be wiser to rely on everyone else. But there’s no reason to isolate Gallup. Any random collection of statistical samplings will include outliers, and just eliminating or underweighting the outliers is not a valid statistical methodology, unless you have reason to believe there’s actually something funky about the outlying samples. Otherwise, the correct approach is to include all samples and let the best guess reflect all of them. ISTM that NS attempted to suggest that there is indeed something funky about the Gallup results, both by picking a few other examples in which Gallup was ostensibly an outlier and by observing some examples in which their polls had a high degree of variance which (he claims) derives from “an endemic issue with their methodology”. Based on this, he derives his “context” which he presents in the form of a “Gallup vs. everyone else” dichotomy. This is what I object to. ---
**Author:** ![Richard\_Parker](https://avatars.discourse-cdn.com/v4/letter/r/35a633/32.png) [@Richard\_Parker](https://boards.straightdope.com/u/Richard_Parker)\ **Post date:** [October 19, 2012, 8:26pm UTC](https://boards.straightdope.com/t/the-weirdness-of-polls-whats-up/638033/84 "2012-10-19T20:26:54Z")
> [@Fotheringay-Phipps](#): > > What I object to is the dichotomy of Gallup vs. “everyone else”. Gallup is part of everyone else, and the average of all polls would include Gallup. If you set up Gallup vs. everyone else, of course you would be wiser to rely on everyone else. But there’s no reason to isolate Gallup. Of course there’s a reason to isolate Gallup! The only way to assess whether a poll is an outlier is to compare it to the average. Is your objection that he would want to label something an outlier? He does so all the time on both sides of the outlier spectrum. The “everyone else” line you quote is him saying " It’s much more likely that Gallup is wrong and everyone else is right than the other way around." That’s what it means to call something an outlier, an assessment with which you seemed to agree above. > [@Fotheringay-Phipps](#): > > Any random collection of statistical samplings will include outliers, and just eliminating or underweighting the outliers is not a valid statistical methodology, unless you have reason to believe there’s actually something funky about the outlying samples. Otherwise, the correct approach is to include all samples and let the best guess reflect all of them. Are you suggesting Nate did anything other than what you describe as the correct approach? If so, I suggest you re-read the article. > [@Fotheringay-Phipps](#): > > ISTM that NS attempted to suggest that there is indeed something funky about the Gallup results, both by picking a few other examples in which Gallup was ostensibly an outlier and by observing some examples in which their polls had a high degree of variance which (he claims) derives from “an endemic issue with their methodology”. Based on this, he derives his “context” which he presents in the form of a “Gallup vs. everyone else” dichotomy. Ok, this objection I understand. You think he has only pointed out a few times when Gallup was wrong, and that this doesn’t conclusively show that Gallup has a bad methodology. In order to establish that reliably, you’d need some kind of pollster rating that compares the relative frequency of wrongness across a large sample. If only someone out there compiled such rating… And, in any event, this is a separate argument from your earlier claim that “Silver is being misleading in that article, in comparing Gallup to ‘the average’” and that “The idea that Gallup is some sort of unique outlier has not been established by anything in that article.” ---
**Author:** ![Chronos](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/chronos/32/134_2.png) [@Chronos](https://boards.straightdope.com/u/Chronos)\ **Post date:** [October 19, 2012, 9:19pm UTC](https://boards.straightdope.com/t/the-weirdness-of-polls-whats-up/638033/85 "2012-10-19T21:19:51Z")
And in any event, it isn’t even necessary to assume that Gallup is doing anything malicious, incompetent, or otherwise wrong. Like I said, outliers happen, and even a polling firm that does everything right will occasionally, just by dumb luck, end up significantly off. Which is what I think is mostly happening here. ---
**Author:** ![iiandyiiii](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/iiandyiiii/32/7924_2.png) [@iiandyiiii](https://boards.straightdope.com/u/iiandyiiii)\ **Post date:** [October 21, 2012, 6:22pm UTC](https://boards.straightdope.com/t/the-weirdness-of-polls-whats-up/638033/86 "2012-10-21T18:22:16Z")
The weirdness continues- Gallup (likely voters) has Romney up by 7, and [Investor’s Business Daily poll](http://livewire.talkingpointsmemo.com/entry/ibd-tipp-poll-obama-expands-lead-over-romney) (also likely voters) has Obama up by 5.7. That’s a _12.7 difference between two different polls_ on the same day. I’m not sure if the Likely Voter models add anything but confusion. ---
**Author:** ![adaher](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@adaher](https://boards.straightdope.com/u/adaher)\ **Post date:** [October 22, 2012, 1:20am UTC](https://boards.straightdope.com/t/the-weirdness-of-polls-whats-up/638033/87 "2012-10-22T01:20:53Z")
Right now the pollsters are just saying, “Um, I really have no idea”. ---
**Author:** ![Fotheringay-Phipps](https://avatars.discourse-cdn.com/v4/letter/f/d9b06d/32.png) [@Fotheringay-Phipps](https://boards.straightdope.com/u/Fotheringay-Phipps)\ **Post date:** [October 22, 2012, 4:55pm UTC](https://boards.straightdope.com/t/the-weirdness-of-polls-whats-up/638033/88 "2012-10-22T16:55:04Z")
> [@Richard\_Parker](#): > > Of course there’s a reason to isolate Gallup! The only way to assess whether a poll is an outlier is to compare it to the average. Is your objection that he would want to label something an outlier? He does so all the time on both sides of the outlier spectrum. > > The “everyone else” line you quote is him saying " It’s much more likely that Gallup is wrong and everyone else is right than the other way around." That’s what it means to call something an outlier, an assessment with which you seemed to agree above. > > Are you suggesting Nate did anything other than what you describe as the correct approach? If so, I suggest you re-read the article. > > Ok, this objection I understand. You think he has only pointed out a few times when Gallup was wrong, and that this doesn’t conclusively show that Gallup has a bad methodology. In order to establish that reliably, you’d need some kind of pollster rating that compares the relative frequency of wrongness across a large sample. If only someone out there compiled such rating… > > And, in any event, this is a separate argument from your earlier claim that “Silver is being misleading in that article, in comparing Gallup to ‘the average’” and that “The idea that Gallup is some sort of unique outlier has not been established by anything in that article.” You’ve split one argument into three. But it’s all one issue. Again in brief, there’s no basis for downgrading the outlier unless you have some reason for believing its methodology is suspect other than the fact of its being an outlier. Otherwise it should be included in the average and weighted as much as anything else. As [Nate Silver](http://fivethirtyeight.blogs.nytimes.com/2012/10/16/falling-prey-to-the-dangerous-temptation-to-cherry-pick-polls/) put it: > [@Nate Silver](#): > > There is a more subtle form of bias, however, that a lot more of us are prone to. That bias is to look at all the data — except for the two or three data points that you like least, which you dismiss as being “outliers.” > > […] > > That is not quite as biased as cherry-picking the best results — but it gets you halfway there, and it is much easier to rationalize. There is something that can be criticized about almost every poll: the methodology, the demographics, the sample size, the pollster’s history or something else. > > Often, these critiques have some truth in them. Not all polls are as methodologically sound as others. But frequently people come up with reasons, valid or otherwise, to avoid looking at the polls they don’t like — while giving a pass to those they do. Wise words. ---
**Author:** ![Richard\_Parker](https://avatars.discourse-cdn.com/v4/letter/r/35a633/32.png) [@Richard\_Parker](https://boards.straightdope.com/u/Richard_Parker)\ **Post date:** [October 22, 2012, 4:56pm UTC](https://boards.straightdope.com/t/the-weirdness-of-polls-whats-up/638033/89 "2012-10-22T16:56:28Z")
Since no one is downgrading the outliers, least of all Nate Silver, I can only surmise you’re retreating from your earlier statements calling his post misleading. ---
**Author:** ![Fotheringay-Phipps](https://avatars.discourse-cdn.com/v4/letter/f/d9b06d/32.png) [@Fotheringay-Phipps](https://boards.straightdope.com/u/Fotheringay-Phipps)\ **Post date:** [October 22, 2012, 4:59pm UTC](https://boards.straightdope.com/t/the-weirdness-of-polls-whats-up/638033/90 "2012-10-22T16:59:08Z")
No. I don’t think he numberically downgrades Gallup in his model, but after a lot of discussion of how much weight to assign them, the rest of his post was to downgrade them as being frequently wrong and erratic etc. ---
**Author:** ![Richard\_Parker](https://avatars.discourse-cdn.com/v4/letter/r/35a633/32.png) [@Richard\_Parker](https://boards.straightdope.com/u/Richard_Parker)\ **Post date:** [October 22, 2012, 5:30pm UTC](https://boards.straightdope.com/t/the-weirdness-of-polls-whats-up/638033/91 "2012-10-22T17:30:58Z")
So you concede that he doesn’t treat Gallup differently in the model. Instead, you meant to use the term “downgrade” to mean criticize? I’m sorry,\*\* F-P\*\*, but your argument is either a moving target or hopelessly unclear. Instead of using your own idiosyncratic definitions for terms like “downgrade,” why don’t you just state your position in plain English. You think it was misleading of Nate to single out Gallup for criticism as an outlier? Is that it? (I honestly don’t think that’s it since you haven’t disputed the fact that he singles out outliers all the time, but it’s the best I can come up with from what you’ve written.) ---
**Author:** ![Fotheringay-Phipps](https://avatars.discourse-cdn.com/v4/letter/f/d9b06d/32.png) [@Fotheringay-Phipps](https://boards.straightdope.com/u/Fotheringay-Phipps)\ **Post date:** [October 22, 2012, 6:14pm UTC](https://boards.straightdope.com/t/the-weirdness-of-polls-whats-up/638033/92 "2012-10-22T18:14:33Z")
> [@Richard\_Parker](#): > > So you concede that he doesn’t treat Gallup differently in the model. Instead, you meant to use the term “downgrade” to mean criticize? No, I think he meant to say that numerically he doesn’t differentiate between Gallup in his model, but he discounts it in suggesting that you should view the Gallup poll as being in a category of its own, distinct from the group of “everyone else”. As he puts it: > [@Nate Silver](#): > > The Gallup poll is accounted for in the forecast model, along with all other state and national surveys. > > However, its results are deeply inconsistent with the results that other polling firms are showing in the presidential race, and the Gallup poll has a history of performing very poorly when that is the case. > [@](#): > > I’m sorry,\*\* F-P\*\*, but your argument is either a moving target or hopelessly unclear. Instead of using your own idiosyncratic definitions for terms like “downgrade,” why don’t you just state your position in plain English. You think it was misleading of Nate to single out Gallup for criticism as an outlier? Is that it? (I honestly don’t think that’s it since you haven’t disputed the fact that he singles out outliers all the time, but it’s the best I can come up with from what you’ve written.) I don’t know what to tell you, I think I’ve been clear enough as well as consistent, and that my understanding of NS’s blog post is correct. It’s not a big enough deal to argue about forever. I don’t have anything to add at this time. ---
**Author:** ![Dob](https://avatars.discourse-cdn.com/v4/letter/d/b2d939/32.png) [@Dob](https://boards.straightdope.com/u/Dob)\ **Post date:** [October 22, 2012, 8:08pm UTC](https://boards.straightdope.com/t/the-weirdness-of-polls-whats-up/638033/93 "2012-10-22T20:08:18Z")
Polls never make sense to me. Obama was at something like 89% just before the first debate and now its down to 69%… based on just that first debate? I dont buy it. He didn’t do well in the debate but its not like he came out and called Mitt a “cracker ass cracker” or something. I dont believe a single debate changes the mood of a nation THAT much. So why they huge spike in such a short period? Were the previous samples doing something different? Did the first debate really turn an Obama voter into a Romney voter? One debate did that? Based on the trending that had to have happened and in large numbers. But I just can’t see how that many Obama supporters before the debate watched it and said, “screw it, i’m voting Romney” Of course the only poll that matters is the one on Nov 6. So go vote!! ---
**Author:** ![Richard\_Parker](https://avatars.discourse-cdn.com/v4/letter/r/35a633/32.png) [@Richard\_Parker](https://boards.straightdope.com/u/Richard_Parker)\ **Post date:** [October 22, 2012, 8:17pm UTC](https://boards.straightdope.com/t/the-weirdness-of-polls-whats-up/638033/94 "2012-10-22T20:17:31Z")
> [@Dob](#): > > I dont believe a single debate changes the mood of a nation THAT much. So why they huge spike in such a short period? Were the previous samples doing something different? Did the first debate really turn an Obama voter into a Romney voter? One debate did that? No. The polls reflect changes in enthusiasm of already-committed voters much more than they reflect switchers or the deciding of the undecided. The first debate popped the Democratic enthusiasm bubble and energized conservatives, making the former less likely to vote and the latter more likely. That was the principal effect. ---
**Author:** ![HurricaneDitka](https://avatars.discourse-cdn.com/v4/letter/h/96bed5/32.png) [@HurricaneDitka](https://boards.straightdope.com/u/HurricaneDitka)\ **Post date:** [October 22, 2012, 8:23pm UTC](https://boards.straightdope.com/t/the-weirdness-of-polls-whats-up/638033/95 "2012-10-22T20:23:08Z")
> [@Dob](#): > > Obama was at something like 89% just before the first debate and now its down to 69%… based on just that first debate? I dont buy it. > > I dont believe a single debate changes the mood of a nation THAT much. So why they huge spike in such a short period? Were the previous samples doing something different? Did the first debate really turn an Obama voter into a Romney voter? One debate did that? I think it was probably a combination of the first debate and Obama’s post-convention bounce fading. ---
**Author:** ![tim314](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/tim314/32/3468_2.png) [@tim314](https://boards.straightdope.com/u/tim314)\ **Post date:** [October 22, 2012, 8:31pm UTC](https://boards.straightdope.com/t/the-weirdness-of-polls-whats-up/638033/96 "2012-10-22T20:31:48Z")
> [@Dob](#): > > Polls never make sense to me. Obama was at something like 89% just before the first debate and now its down to 69%… based on just that first debate? I dont buy it. He didn’t do well in the debate but its not like he came out and called Mitt a “cracker ass cracker” or something. One thing to keep in mind is “how likely is he to win” and “how much of the country supports him” are two different questions. Suppose, just as an example, that Obama was supported by 51% of likely voters. But suppose also that the uncertainty on that number was very, very small, such that we were 99% sure Obama had more than 50% support. Then we could say\* that Obama had a 99% chance of winning the election (if it were held today). Then the debate happens, and it shifts 2% of the electorate to Romney, so now we’re 99% sure Romney would win the election (if held today). This is a very exaggerated, unrealistic example, but it illustrates a point, which is that a small shift in the electorate can sometimes result in a big shift in the likelihood of victory. On the flip side, if 90% of the voters supported Obama then a huge drop, to say, 60% support, wouldn’t do much to change the likely outcome of the election. - ignoring the difference between popular vote and electoral college, and ignoring any questions about what “likely voters” means or how good we are at identifying them ---
**Author:** ![Dob](https://avatars.discourse-cdn.com/v4/letter/d/b2d939/32.png) [@Dob](https://boards.straightdope.com/u/Dob)\ **Post date:** [October 22, 2012, 8:43pm UTC](https://boards.straightdope.com/t/the-weirdness-of-polls-whats-up/638033/97 "2012-10-22T20:43:35Z")
> [@Richard\_Parker](#): > > No. The polls reflect changes in enthusiasm of already-committed voters much more than they reflect switchers or the deciding of the undecided. The first debate popped the Democratic enthusiasm bubble and energized conservatives, making the former less likely to vote and the latter more likely. That was the principal effect. I get that to extent, but the polls dont poll un-likely voters. So pollsters are now callign folks that a month ago were all Obama, but now are …Romney? Are they just calling different people, and the new people dont like Obama?? Enthusiasm would hinder the actual voting, but enthusied or not if someone asks who you are voting for why wouldnt you say “Obama” if you said the same thing 4 weeks ago?? my head, it hurts… ---
**Author:** ![HurricaneDitka](https://avatars.discourse-cdn.com/v4/letter/h/96bed5/32.png) [@HurricaneDitka](https://boards.straightdope.com/u/HurricaneDitka)\ **Post date:** [October 22, 2012, 8:49pm UTC](https://boards.straightdope.com/t/the-weirdness-of-polls-whats-up/638033/98 "2012-10-22T20:49:45Z")
> [@Dob](#): > > Enthusiasm would hinder the actual voting, but enthusied or not if someone asks who you are voting for why wouldnt you say “Obama” if you said the same thing 4 weeks ago?? > > my head, it hurts… That’s where the difference between “Likely Voter” and “Registered Voter” comes into play. If a pollster called you right after the Dem convention and asked you who you were voting for and how enthused you were to vote, and your answer was “Obama, and 100%” then they put you in their results as a “Likely Voter”. A few weeks later, Obama just got hammered in Debate #1 and the pollster calls you back. “How enthusiastic are you to vote?” and now you feel a little glum and so you say, “um … i don’t know, not very”. Now you’re not considered a “Likely Voter”, so the polls of “Likely Voters” shows a swing towards Romney. ---
**Author:** ![Terr](https://avatars.discourse-cdn.com/v4/letter/t/839c29/32.png) [@Terr](https://boards.straightdope.com/u/Terr)\ **Post date:** [October 22, 2012, 8:50pm UTC](https://boards.straightdope.com/t/the-weirdness-of-polls-whats-up/638033/99 "2012-10-22T20:50:05Z")
> [@Dob](#): > > Polls never make sense to me. Obama was at something like 89% just before the first debate and now its down to 69%… based on just that first debate? I dont buy it. He didn’t do well in the debate but its not like he came out and called Mitt a “cracker ass cracker” or something. > > I dont believe a single debate changes the mood of a nation THAT much. So why they huge spike in such a short period? Were the previous samples doing something different? Did the first debate really turn an Obama voter into a Romney voter? One debate did that? > > Based on the trending that had to have happened and in large numbers. But I just can’t see how that many Obama supporters before the debate watched it and said, “screw it, i’m voting Romney” What the polls show is quite possible. Your mistake is thinking that those were “Obama supporters” that switched their opinion. In fact, those were most probably people who were going to vote for Obama because he was a known thing, with only a very lukewarm level of enthusiasm, and who needed some idea of what Romney was like in order to switch their “support”. ---
**Author:** ![Richard\_Parker](https://avatars.discourse-cdn.com/v4/letter/r/35a633/32.png) [@Richard\_Parker](https://boards.straightdope.com/u/Richard_Parker)\ **Post date:** [October 22, 2012, 8:50pm UTC](https://boards.straightdope.com/t/the-weirdness-of-polls-whats-up/638033/100 "2012-10-22T20:50:55Z")
> [@Dob](#): > > I get that to extent, but the polls dont poll un-likely voters. So pollsters are now callign folks that a month ago were all Obama, but now are …Romney? Are they just calling different people, and the new people dont like Obama?? > > Enthusiasm would hinder the actual voting, but enthusied or not if someone asks who you are voting for why wouldnt you say “Obama” if you said the same thing 4 weeks ago?? > > my head, it hurts… No, it works like this: Pollsters call a whole ton of people, and only the ones who pass the likely voter screen get counted. A pollster calls and asks “How likely are you to vote in November?” They might give you some options, ranging from not at all likely to definitely voting. If you’re an Obama voter after the first debate, maybe your response shifted from “definitely voting” to “pretty sure I’ll vote.” Conversely, if you’re a Romney voter, you might have gone from “somewhat likely” to “I already voted!” These shifts affect the likely voter numbers quite a bit. Relatedly, you might just be more interested in participating in a poll in the first place (instead of hanging up on them) if you’re feeling enthusiastic about the election. That phenomenon doesn’t entirely explain Romney’s bounce. But it explains a very large part of it. [Previous page](https://boards.straightdope.com/t/the-weirdness-of-polls-whats-up/638033.md?page=4) [Next page](https://boards.straightdope.com/t/the-weirdness-of-polls-whats-up/638033.md?page=6)