The new Pew Political Typology poll has the political blogs putting out the usual punditry about the state of American politics. But if you look at the poll itself, the design is awful. It’s constructed as a series of “which comes closer” questions–which is fine–but a majority of the questions aren’t even on two ends of the same continuum. For example:
This is a statement of principle paired with an empirical statement, and not only are they different in kind, they aren’t even opposites. It is perfectly possible (and in my view, correct and accurate) to agree equally with both statements. Or consider this abomination of logic:
It is possible (and indeed, true, in my opinion) for most hard workers to be able to make it, but for hard work and determination to be no guarantee for most people. They are not mutually exclusive statements, and neither is “closer” to the truth.
The best you can say for this kind of survey design is that people who tend to be X tend to pick Y answer. And that’s fine if your goal is just to classify the poll-taker into a pre-defined category. But that’s not what Pew purports to do! They also purport to be determining what these categories believe at the same time. For example, consider this binary:
Now, I don’t think either of these is true. If forced to choose, I might pick 2 just because I know that’s the “liberal” answer, and because I resent the “without doing anything in return” part of the first question. But if you then concluded that liberals believe “Poor people have hard lives because government benefits don’t go far enough” you’d be wildly mischaracterizing my view and that of many liberals (and the same is true for those picking the conservative answer). But that’s exactly what Pew, and the many reporters using this poll for their stories, have done.
In designing a survey the trick is to make both options equally appealing. That way you can get a approximation of what the person thinks and not what the person thinks you want them to say. The survey seems to have done a pretty good job doing this. Putting people into distinct groups only makes sense if there are discreet differences between the groups, I don’t see that there is, but I could be missing something.
What you can’t use this for is like an opinion poll. If 50% of people pick option A then you can’t say option A is actually what 50% of the people think. Lots of people are ignorant about survey design and usage so you can bet lots of ignorant things will be said about the results.
It looks to me like they have tried to correlate the answer on certain questions with results of their larger political typology poll. Kind of like the way the statisticians at OKCupid found that compatibility in a relationship can be boiled down to how you answer 3 questions:
Do you like horror movies?
Have you ever traveled around another country alone?
Would it be fun to give up everything and go live on a sailboat?
It does not matter, statistically at least, how you answer any other question. Even questions about religion or morality don’t really correlate with compatibility like these three questions do.
Presumably the folks at Pew have done something similar. Whether your answers really tell someone anything about your Typology may be beside the point, if you answer these questions a certain way it statistically places you within a group.
Who knows though? I could be wrong. I do know that the last time they did this it was awful. I went through the quiz several times and it was seemingly impossible, no matter how you answered, to place yourself in a particular group. I did not really agree with the group i was placed in when running though the types and thought my opinions were more aligned with other groups. This year I agree with where they put me.
Bear in mind, Pew has been doing this every few years since 1987 (and every time they do it, they come up with a new set of typologies). I’m sure they’ve learned a few things about survey design over the years.
How is it useful to change categories every year? You’d think it’d be more valuable to keep mostly the same categories year to year, so you can track what’s happening across time. BrainGlutton’s quote seems to be trying to do this, but the “business conservatives”, say, aren’t the same thing as the “enterprise Republicans”.
What you’re missing is that you don’t have a clue how surveys are designed.
First–the first pair you mentioned, any traditional conservative and every libertarian will reliably pick the second one, and any traditional liberal will reliably pick the first one.
The second pair is slightly iffier, but most conservatives and libertarians will pick the first statement, with most liberals picking the second statement.
The third pair is also pretty clear-cut; most conservatives and libertarians pick the first, and all liberals pick the second.
Those three pairs of statements are sufficient in and of themselves to accurately categorize the broad political views of most politically active adults. Anything else is simply refining the picture.
Edit: I conducted telephone surveys for over a year. The listed examples are perfectly normal, and widely used by multiple companies. As a matter of fact, I probably did at least one survey that had those three pairs word-for-word.
Aren’t these two mutually exclusive? I don’t know what your complaint is.
1 says that, for most people, if they work hard, it’s a guarantee they’ll succeed.
2. says that for most people, if they work hard, there’s no guarantee they’ll succeed.
Next Generation Left (even though I’m mid-fifties). I didn’t answer questions 4, 12, and 14. I wonder if no answer was allowed in the original survey. If not answering is intended to be allowed, then they should state that. Otherwise it is unfair, in that only intelligent people (mostly liberals) would consider the possibility of no response.
ETA: Yes, I think this is a horrible design. I’m sure there’s some liberal, newfangled justification to it, but it just a’int right darn it.
I answered down-the-line left on the economics issues, environmentalist on the environmental issues, and isolationist on foreign policy, but the social questions were tough. What happens if you favour immigration from some parts of the world but not others? What do you say if you don’t think the problems of African Americans are their own fault OR due to racism? How do you respond if you don’t especially care about the marriage rate, but care a lot about the fertility rate?
is correct. Their problem is that that’s true, which makes it very hard to classify people.
They’ve chosen their categories and they match people to them. They classified me as “Next Generation Left”. When I look at their “Ideological Placement”, I came in above “Mixed” with the bulk of my category to my left. So not a good fit. But it’s nice to see the tails of their categories; it’s a good indicator of the fuzziness of their groups. What I’d really like to see are numbers about how well I fit into each group, instead of a single “this is your best fit”.
The trouble, as has been pointed out, is that the questions have binary choices. There’s no weighting based on priority or intensity. They have no way to distinguish the cases of 1) I strongly agree with statement A and disagree with statement B vs 2) strongly agree with A and somewhat agree with B vs 3) disagree with A and strongly disagree with B. I pick A in each of those cases, but my beliefs are very different.
I’m not even really complaining about the use of the poll to place people into pre-defined buckets. That has some issues but seems more or less methodologically sound.
My chief complaint is that they then purport to say what those buckets believe, based on the answers to the binary questions. I see no basis at all for that conclusion. All you can reliably conclude is that X group thought Y answer “came closest” to their belief. It is just a logical travesty to conclude that X group believes Y (or that Z% of X group believes Y).
That makes nonsense out of all the reporting about how conservatives think the poor have it easy. The people reading the poll that way, including Pew, don’t have a leg to stand on.
I agree with Mr. Parker. Someone referred me to the quiz, ands I found that, as often as not, I agreed with both statements or disagreed with both. I gave up the quiz long before I got to the finish, because I couldn’t see the point in continuing.
James Madison said it better than I could: "If men were angels, no government would be necessary. If angels were to govern men, neither external nor internal controls on government would be necessary. In framing a government which is to be administered by men over men, the great difficulty lies in this: you must first enable the government to control the governed; and in the next place oblige it to control itself. A dependence on the people is, no doubt, the primary control on the government; but experience has taught mankind the necessity of auxiliary precautions."
Is that a liberal or conservative philosophy?
Would the man who wrote that be able to choose blithely between “People should be free to do what they want without government interference” and “People need strong government”?
He was wrong about that, of course. Many things make government necessary that have nothing to do with the human tendency to misbehave. E.g., we need it to provide many public services other than policing.
