Problems with polls

[DSeid]

I can’t find a link to the radio bit I heard today that made me want to post this but the same points are made here: multiple factors have made polling less meaningful now than it was even just a few years ago and aggregation does not solve the problems.

… Election polling is in near crisis, and we pollsters know. Two trends are driving the increasing unreliability of election and other polling in the United States: the growth of cellphones and the decline in people willing to answer surveys. Coupled, they have made high-quality research much more expensive to do, so there is less of it. This has opened the door for less scientifically based, less well-tested techniques. To top it off, a perennial election polling problem, how to identify “likely voters,” has become even thornier. …

… Those paying close attention to the 2016 election should exercise caution as they read the polls. Because of the high cost, the difficulty in locating the small number of voters who will actually turn out in primaries and the increasing reliance on non-probability Internet polls, you are likely to see a lot of conflicting numbers. To make matters still worse, the cellphone problem is more acute in states than it is at the national level, because area codes and exchanges often no longer respect state or congressional boundaries. Some polling organizations will move to sampling from voter lists, which will miss recently registered voters and campaigns’ efforts to mobilize them.

We are less sure how to conduct good survey research now than we were four years ago, and much less than eight years ago. And don’t look for too much help in what the polling aggregation sites may be offering. They, too, have been falling further off the track of late. It’s not their fault. They are only as good as the raw material they have to work with.

In short, polls and pollsters are going to be less reliable. We may not even know when we’re off base. What this means for 2016 is anybody’s guess.

So anybody want to take a guess?

[The_wind_of_my_soul]

I have found it interesting to see the alternate ways that bloggers, journalists, etc. attempt to gauge public support for a candidate. I’ve seen graphs showing the number of Google searches, articles comparing the big donor support for various candidates, articles on the candidate’s abilities to raise money in small donations, articles on the size of the turnout for rallies, etc. One of the strangest things I see is journalists almost arbitrarily deciding who they consider a big contender. They’ll talk about candidates that are polling fairly well (comparatively), such as Bush and Rubio, and then throw in the name of some other candidate who’s polling poorly, but for some reason the journalist takes the candidate seriously – names like Kasich, Christie, and even Graham.

My guess is that Cruz will be the biggest surprise. Cruz’s super-PAC has outfundraised every Republican candidate except Bush, and yet his public persona is anti-establishment. I think this puts him in a unique position: his grassroots supporters will be excited enough to not only vote for him in polls, but actually vote for him when it matters; his big backers may have enough influence to drum up his appeal to a broader audience, so that the people who don’t participate in polls but do eventually vote in elections will also vote for him.

[DSeid]

It does get me reading the methods more. I feel for the polling companies. They have a huge demand for product to satisfy the political entertainment industry yet less and less ability to create meaningful product.

Do you do a combo of landlines and cells and “correct” the sample to actual demographics? (Most recent poll today is Monmouth and that is their approach.) Use a hybrid landline/internet approach? (Next most recent PPP did that.)

I have a landline but it really only gets used for calls out as my house has a few dead zones; I never answer a number on it that I do not know. And rarely would on my cell. I doubt my behavior is so unique nowadays and I doubt it is completely randomly distributed. Yeah I know it sounds like “skewed polls” but the thing is no one can reasonably predict which sorts of ways it would skew!

Whichever candidates win or lose the initial states I think I am now more interested in seeing how the polling (including the poll aggregators) performs, even up close to election day. I am betting there will be some major polling fails. 538 and my preferred Sam Wang may be already past their peak times and the pundits may rise again! (Please reassure me it aint so.)

[glowacks]

I have for quite some time wondered myself who is it that actually answers pollsters’ questions. I have absolutely no desire to take the time to answer a bunch of questions with no benefit to me, and I don’t know many people who would have such a desire. There’s no easy way to force someone to answer questions if they don’t want to. With caller-ID, no one answers their phone unless they know who is calling, fully expecting anyone with anything important to say will leave a message. And thus there seems to be a real problem when it comes to trying to determine whether you have a fair sample or not; while you can correct for any particular factor that you know the true distribution for, how exactly are you supposed to figure out those distributions for arbitrarily many variables and associated reasons that people aren’t answering your polls?

[guizot]

@glowacks

I have for quite some time wondered myself who is it that actually answers pollsters’ questions. I have absolutely no desire to take the time to answer a bunch of questions with no benefit to me, and I don’t know many people who would have such a desire.

Isn’t it generally elderly people who tend to be retired and have more time on their hands? And younger people are generally more likely not to have a landline. I guess this is one way that they may be skewed.

[Peremensoe]

@glowacks

I have absolutely no desire to take the time to answer a bunch of questions with no benefit to me, and I don’t know many people who would have such a desire.

No benefit? You get to represent your views. You’re ‘voting’ in a much smaller electorate, when you answer a political survey, than when you cast your actual vote.

[DinoR]

but… but… but … numbers. Numbers must mean something.

Nice catch. I’ve seen little bits related (mostly concerns with the internet polls) but nothing like the overall issues with polling going forward.

I’d guess were a couple election cycles from the reality fully hitting home for those not either in the polling industry or data analysts for campaigns. They’ll be some room to spin/ignore small failures as the response rate continues to drop. Eventually reality hits home. It will be interesting to see what effects it will have on politicians as they realize there’s lower data reliability.

[RTFirefly]

@guizot

Isn’t it generally elderly people who tend to be retired and have more time on their hands? And younger people are generally more likely not to have a landline. I guess this is one way that they may be skewed.

One thing I wonder about landlines v. cellphones is, why are cellphones still treated as special snowflakes? Fifteen years ago, this made sense: practically everyone’s contract allowed a certain number of minutes each month, and if you exceeded those minutes, the rate went way up. Now I’d guess that the vast majority of cell phone users have unlimited voice and text. And surely there has to be a way to set things up for the caller to pay for both ends of the call anyway.

[Chronos]

I think my plan still has limited minutes, but it’s so many of them that I’ve never even come close to hitting those limits.

[DigitalC]

I haven’t seen any problems with polls giving accurate predictions for elections so far.

[steronz]

@RTFirefly

One thing I wonder about landlines v. cellphones is, why are cellphones still treated as special snowflakes? Fifteen years ago, this made sense: practically everyone’s contract allowed a certain number of minutes each month, and if you exceeded those minutes, the rate went way up. Now I’d guess that the vast majority of cell phone users have unlimited voice and text. And surely there has to be a way to set things up for the caller to pay for both ends of the call anyway.

Shhh, why do you want to ruin it for everyone?

[DSeid]

@DigitalC

I haven’t seen any problems with polls giving accurate predictions for elections so far.

Haven’t been paying much attention then.

The U.K. elections - major prediction fail. The Israeli election also. Polls in the lead up to the Scottish independence vote? Way off. Mid terms? The polls failed to see the GOP landslide coming.

As Nate Silver put it after he, along with everyone else, failed so horribly in the Scottish referendum: “The World May Have a Polling Problem”

@Nate Silver

In fact, it’s become harder to find an election in which the polls did all that well.

Consider what are probably the four highest-profile elections of the past year, at least from the standpoint of the U.S. and U.K. media:

The final polls showed a close result in the Scottish independence referendum, with the “no” side projected to win by just 2 to 3 percentage points. In fact, “no” won by almost 11 percentage points.
Although polls correctly implied that Republicans were favored to win the Senate in the 2014 U.S. midterms, they nevertheless significantly underestimated the GOP’s performance. Republicans’ margins over Democrats were about 4 points better than the polls in the average Senate race.
Pre-election polls badly underestimated Likud’s performance in the Israeli legislative elections earlier this year, projecting the party to about 22 seats in the Knesset when it in fact won 30. (Exit polls on election night weren’t very good either.)
At least the polls got the 2012 U.S. presidential election right? Well, sort of. They correctly predicted President Obama to be re-elected. But Obama beat the final polling averages by about 3 points nationwide. Had the error run in the other direction, Mitt Romney would have won the popular vote and perhaps the Electoral College.

Perhaps it’s just been a run of bad luck. But there are lots of reasons to worry about the state of the polling industry. Voters are becoming harder to contact, especially on landline telephones. Online polls have become commonplace, but some eschew probability sampling, historically the bedrock of polling methodology. And in the U.S., some pollsters have been caught withholding results when they differ from other surveys, “herding” toward a false consensus about a race instead of behaving independently. There may be more difficult times ahead for the polling industry.

Hey I will still discuss the data I have even while I mourn the lack of the data I want. But Nate Silver may be right here that he will not be so often right again.

[Kent_Clark]

@glowacks

I have for quite some time wondered myself who is it that actually answers pollsters’ questions.

I do, but I’m a cranky old man who also writes letters to the editor!

By the way, I also study and vote in every single election – bond issues, library taxes, school boards, you name it.

Now get the hell off my lawn.

[bup]

@guizot

Isn’t it generally elderly people who tend to be retired and have more time on their hands? And younger people are generally more likely not to have a landline. I guess this is one way that they may be skewed.

Well, they ask pollees their ages, and then artificially weight the data to rectify the problem you’re talking about. The problem is back to square one - just getting *enough *data to make decent inferences.

Stats from other sources, like The Wind suggests in post 2, are helpful to measure enthusiasm, but most of the numbers aren’t at the state level. The presidential election, of course, is all about which states you win.

It’s a real problem. Maybe in ten years’ time, texting will be ubiquitous enough that pollsters can do polling that way. We’re blah-blah, please click this link and answer a few questions if you can in the next 72 hours.

[RickJay]

I’ve been hearing these same things for years, and yet in 2012 the polling aggregator masters, like Nate Silver, predicted the election bang on. And I say they’ll do it again in 2016.

[That_Don_Guy]

@DigitalC

I haven’t seen any problems with polls giving accurate predictions for elections so far.

A lot of people would mention Ohio, 2004. (A lot of other people would respond, “Diebold” - Senator Boxer even challenged the vote in Congress - but nobody ever did find a Diebold voting machine in Ohio that didn’t leave a printed result of every vote.)

One theory is, do more people in certain parts of the country vote more than they used to? In 2014, a number of races had the Republican ahead for most of the night, but invariably the last areas to report were heavily populated urban ones, and the Democrats won the seat.

[RTFirefly]

Maybe they will. But I wouldn’t put any money on 2018.

Falling response rates are also causing concern to those of us who [del]design and arrange shrubberies[/del] work on governmental demographic surveys. Even surveys that pay the money it takes to get around the landline/cellphone problems by sticking with more expensive face-to-face interviews are seeing response rates drop, year after year after year.

The real problem is, the larger your proportion of nonrespondents, the harder it is to tell if your respondents accurately represent them - and the more it matters if they don’t.

[DSeid]

@RickJay

I’ve been hearing these same things for years, and yet in 2012 the polling aggregator masters, like Nate Silver, predicted the election bang on. And I say they’ll do it again in 2016.

Well given what Nate Silver’s prediction is: “more difficult times ahead for the polling industry”, yeah.

[Fuzzy_wuzzy]

@RickJay

I’ve been hearing these same things for years, and yet in 2012 the polling aggregator masters, like Nate Silver, predicted the election bang on. And I say they’ll do it again in 2016.

Here in the UK we just had a General Election that the pollsters got badly wrong, even analysts such as Nate Silver and his team. Admittedly the rise and fall of relatively minor parties made this election harder to forecast, but the outcome was still an acute embarrassment for pollsters. I doubt your pollsters in the US are much superior to ours in the UK. What happened in the UK may just be a result of poor methodology, but it may also be a prediction of things to come in the accuracy of the polling industry.

[up_the_junction]

@DSeid

Haven’t been paying much attention then.

The U.K. elections - major prediction fail. The Israeli election also.

Certainly in the UK and, I think Israel, prediction polls are unable to calibrate the effect of tactical voting - that is voting to keep a candidate out. Or even protest voting for a different candidate.

This has been significant in areas of the UK

It basically makes the whole trade of polling a bogus con, but the newspapers love a simple graphical image and everyone wants that on going pre-election narrative.

Polling seems to work better in binary races IF - and this fails to happen surprisingly often - the pollsters ask the right question/s.