The PIPA Report: Americans on Iraq on the Eve of the Presidential Election

Great Debates
1

In this thread, elucidator calls out Bill H. to substantiate his critique of the PIPA study. Bill claims that the study is “not non-biased”, “was designed to be a tool for the discrediting of President Bush”, and “lacked scientific credibility on several fronts.”

I believe that this critique is without merit and smacks of vague, repeated dogma. However, the PIPA survey is highly problematic. Since it has been cited frequently here, I would like to venture my analysis for general consumption. Perhaps elucidator and Bill H. will have something to add.

A word about my own inclinations and qualifications. I was indeed a Kerry voter, and am an agnostic New Yorker. I suppose that this makes me a godless, elitist liberal. I also have a graduate degree in political science, and I specialized in quantitative methodology. In other words, I studied human political behavior by creating mathematical models and tested them using “classical” statistical tools. I am not an international expert in my field, nor am I a professor with decades of experience. I have just seen this type of thing before and even done some of it myself.

The problem with this study is that it is fundamentally uninformative.

The goal of most voting behavior models is to figure out how certain quantitative measures of voter characteristics (covariates or independent variables, terms that I use interchangeably) affect a dependent variable, which is often either voter turnout or candidate choice. A political scientist is supposed to have some sort of prior belief as to how the covariates affect the dependent variable and constructs an equation, or a model, to represent this effect. How significant the effects actually are is determined by the statistical testing.

2

Goddamn, that should not have happened. Meant to preview.

Continuing…

So the problem with this study is that it merely evaluates a correlation between the erstwhile failure of the American public to absorb information and a particular electoral behavior. Fine, this is pretty straightforward. An eighth grader can calculate the correlation between a vote for Bush and ignorance of the Duelfer report. The study’s fifteen some-odd pages of findings are very tedious, and the graphics aren’t even very good.

The real problem with this study is not in its argument but in its implicit suggestion that you may use this sample to somehow draw inferences about the population. The researchers frame this study in a series of questions precipitated by “How do Americans feel about…”, and suggest that they can answer these questions with their study. They try to infer the effects of ignorance on electoral behavior.

This is incorrect and misleading. The researchers are essentially arguing by apposition: if a bunch of Bush voters were ignorant, then the rest are, too.

It is very possible to make this claim and it would not be difficult to design a study to test it. The quantity of interest is the voter’s electoral behavior. It is possible to present a mathematical model which argues that ignorance, which can be quantified, affects a voter’s presidential choice. One can test this model with a sample, and report the magnitude of the effects (the coefficients of the independent variables), the standard errors, and the statistical significance. However, any time one chooses a mathematical model, the modeler has to make certain assumptions about the real world. Sometimes these are reasonable, sometimes less so. Often if the assumptions are relaxed, the model falls apart entirely. Often models can be criticized on the grounds of endogeneity or omitted variable bias. These are very serious and can undermine the inferential value of a model.

The authors are trying to answer a question about the American population without actually presenting a model for scrutiny. This abrogates them of the responsibility of presenting coefficients, standard errors, and confidence intervals. It also prevents the assumptions of their model, which usually have serious real world implications, from being challenged. Their correlations offer us no way whatsoever to predict how a voter of certain characteristics will behave. The only statistic they report, the so-called “margin of error”, is thoroughly uninformative. Finally, to my irritation, they do not even include the data set. I would have loved the opportunity to run some of the numbers myself.

The study is not unhelpful because it is somehow politically biased or unscientific. Knowledge Networks is an excellent outfit, and I believe they conduct these surveys with considerable professional integrity. The study is garbage because all it tells us is how 968 respondents answered a survey. This is in no way informative about the state of the American electorate.

I would be happy to go into more technical detail if anyone wants. I do not want anyone to think that I am just handwaving. If you are satisfied with my analysis, then there is no point in making myself out to be one of those condescending, pseudo-intellectual northeasterners. Alternatively, you can check out this Wikipedia article that should give you a nice summary of statistical inference and theory.

If you are interested in more technical pieces on binary dependent variable models, the literature is huge. I would be happy to suggest some titles.

3

Thanks, Maeglin, for your efforts.
The first thing that leaps out at me is this, however:

I really don’t know what the researchers were arguing, but what I’ve been dragging the study out to argue is this: There are many ignorant voters. Of that group, the majority voted for Bush. By an extremely wide margin.

Many voters are NOT ignorant. Of those, the majority voted for Kerry. By a wide margin.

Both the ignorant and the informed agree that the Bush administration is feeding them the information that is wrong. However, the ignorant people believe it, and the non -ignorant people do not.

Hence, more informed people see the reality and choose Kerry, ignorant people choose Bush.

Which is a different argument/conclusion.

4

This is not material. The problem with the study is that it is impossible to infer from the sample anything about the population. You can argue any conclusion you like from the study, but the researchers did not use any of the numerous statistical tools at their disposal to attempt to bridge the gap between sample and population. That is the most serious error. The result is that intelligent people are trying to bridge that gap for the researchers.

They are incorrrect.

5

Maeglin:

Excellent post.

I had tried to explain these flaws in other threads, and ended up babbling simplistically, offering analogies that were half-baked and failed to convince, because I didn’t quite grasp the detail myself. I appreciate your cogent and thorough explanation.

Stoid, as she does whenever a fact conflicts with her worldview, is going to cling to the worldview and insist the fact must be wrong. I doubt any amount of detailed explanation on your part will change that.

HOWEVER – don’t let that dissuade you from continuing to explain and persuade. Because Stoid is one person, and there are many readers who never post but who read, and learn. It’s for their benefit that continued explanations are useful, and believe me: they thank you for it.

6

I just love this place sometimes.

First, thanks Maeglin for the informative post.

Second Maeglin says:

“The study is garbage because all it tells us is how 968 respondents answered a survey. This is in no way informative about the state of the American electorate.”

To which Stoid replies:

“I really don’t know what the researchers were arguing, but what I’ve been dragging the study out to argue is this: There are many ignorant voters. Of that group, the majority voted for Bush. By an extremely wide margin.”

Simply amazing Stoid. Your arguement seems to come down to this: “Bush voters are ignorant because this study says so even though I do not know what the study means or how the results should be interpreted.”

Bravo.

Slee

7

I don’t really have anything substantial to add, but I’ve seen this PIPA report tossed around in several threads to explain why Bush voters are stupid (well, substantially less informed than Kerry voters I suppose), etc. Thanks for the great explaination of it Maeglin.

Carry on. :slight_smile:

-XT

8

Yeah, that would be me. I was just sitting here, watching this porno loop I’ve got of hot lesbo action with J Lo and Angelina J. and I thought, Nah, this is pretty boring, what I really need is some binary dependent variable models…

OK, couple of things. First, the question of methodology and intent. Maeglin criticizes the study for reasons that I wouldn’t understand to save my soul. Fine, I’ll take his word for it, until a bigger smarty-pants says otherwise. But it seems, dimly, I admit, that technically one cannot infer from the sample that Bush voters are of a kind. OK, no sweat. One can only derive that the voters sampled were ill informed.

Also, one can’t necessarily infer that the Bush voters voted the way they did because they were ill-informed, only that they were ill-informed and they voted a certain way, but the vote is not necessarily a result of the ignorance.

Well, OK, but so what? I’m not trying to suggest that if the Bush voters knew the facts of which they were ignorant, they would have voted differently. They might, they might not. One might certainly hope so, but that lies outside of matter at hand, no? All we know is that this sample was a) Bush supporters who b) were ignorant of certain crucial facts.

What I’m ineptly getting at is: while it may not be correct, according to statistical science, to state that Bush voters are ignorant, it may nonetheless be reasonable to infer same, unless we have some reason to believe that this sample of voters is exceptional. Note I don’t say this proves it, only that it is a reasonable inference. That which is reasonable and that which is proven aren’t always the same thing.

Secondly, I’m assuming that Maeglin speaks from standard orthodox statistical science. I’m assuming that because I haven’t a clue, and he seems like a nice enough fellow. (Which is why I own a '76 Dodge Dart with a tranny made of oatmeal, but nevermind…) Now, I assume that the authors of this PIPA thingy are members of the same Statistical Orthodox Church. That what Maeglin tells us is not news to them.

So…were they trying to pull a fast one? Or is an inference of Bush voter ignorance a reasonable inference, if not a scientificly valid inference?

9

The key here is the phrase “representative sample.” In other words not the sample quantity, but the sample quality. It is mathematically sound to draw conclusions about large groups from relatively small samples. The University of Michigan Survey Center(considered one of the best surveying outfits outside of the Census Bureau) draws their famous consumer confidence index from a sample size of 500 each survey period. In general, random sampling is considered to have a high degree of probability of producing a representative sample, or at least one of the highest probabilities.

The Census Bureau rehashes this every decade. There are always arguements for using sampling and mathematical formulas to determine the census counts and data as opposed to the mass mailing of millions of forms and all that goes along with it. The Census itself works with representative data. Roughly one out of seven homes gets “the long form” during a census cycle. Those 1/7ths of the population are the only hard data collected for dozens and dozens of data points the Census publishes as “Nationwide” averages. They are, in reality, the averages of all those 1/7ths of the nation who got the long forms(and bothered to turn them in, which is an even smaller number).

So the question is not really how many were in the sample, it is the confidence level that the sample was representative. According to the PIPA report itself the methodology to find respondents was thus.

I’m not alarmed at this methodology. It seems perfectly in line with industry standards. Random samples, nationwide. The larger a sample the smaller the margin of error(and the more likely you are to have a representative sample) but the extrapolation to general population is not necessarially fallacious. I’d say there is a better chance of it being representative of the larger population than not.

Now there are other ways to bias a survey. All kinds of irregularities, from question phrasing, to question order, to time of day called(people tend to be cranky during dinner and may not give their usual answer). But the sample size itself is not necessarially problematic. Nationwide polls/surveys are conducted all the time with this number of respondents or less. I already mentioned the University of Michigan Consumer Index, but the Gallup organization has a current “Gallup Glance” on the bottom of their home page right now about confidence in government to handle domestic and international issues.

[quote]
Roughly 6 in 10 Americans say they have at least a fair amount of trust and confidence in the federal government to handle international and domestic problems. The sample methodology/size are bolded.

Enjoy,
Steven

10

It would have been very easy to design the study to test the claim that you now want us to infer.

I would argue that the fact that the researchers knew this, and deliberately decided not to test it, invites a “fast one” inference.

11

No kidding? That’s a shocker. You would, huh? Its not because the recent crushing landslide of a mandate proves it, is it? Because I’m not sure that is entirely equivalent with a lubricious binary modality. I mean, you pretty much thought that anyway, right, even before anyone ventured to advise us?

12

I see Mtgman wrote what I was about to, and much better than I would have.

I’m not a statistician, but…

… that seems to be a criticism of surveying in general.

If you were to write a survey to test how well ignorance of certain facts correlates with voting for a certain candidate, what would be different about it? A larger sample, a sample chosen differently, different questions, more statistical details listed in the final report?

Would you mind explaining these numerous tools for the less statistics-minded among us?

13

On further reflection, I would like to withdraw this suggestion and inference. I believe this discussion is potentially a rarity these days: a honest, non-partisan debate about statistical tools and quantitative methodology, with an expert generously donating time to explain things to laity in the field. My comment may help derail this into another slug-fest, and I’d rather not be part of the problem.

14

Let us propose this as our null hypothesis: “Bush and Kerry supporters believed factually incorrect statements equally.”

Now, if Bush and Kerry supporters were as deluded as each other, we could expect a similar probability of freak results showing Kerry supporters massively more deluded than well-informed Bushites. Thus, we are actually looking at a two tailed test.

We must ask “what is the probability of getting the results we did if this hypothesis is true?” In other words, how unlikely a poll must we have carried out to get freak results showing the null hypothesis was false even though it was actually true? The traditional scientific threshold is significance at 5%, and the rigorous threshold is significance at 1%. For a null hypothesis of “The coin is unbiased”, then for 10,000 flips, more than 51.3% heads or less than 48.7% heads would require us to reject the null hypothesis under the 1% threshold.

No matter which numbers from the PIPA report you plug in, the conclusion is inescapable:

The null hypothesis must be rejected, even under the rigorous 1% threshold.

Bush supporters were vastly more likely to believe incorrect statements than Kerry supporters. Were this a medical trail, the question would be not whether there was a difference bwteen the two groups, but what the cause of the difference was.

Any explanations from Bush voters as to the cause of these statistically obvious differences?

15

Let’s propose it as “Bush and Kerry supporters believed these particular factually incorrect statements equally.”

16

Yes Bricker, agreed.

And in the accompanying conclusion: Bush supporters were vastly more likely to believe these incorrect statements than Kerry supporters. Were this a medical trial, the question would be not whether there was a difference between the two groups, or whether the sample space was big enough to represent the two groups satisfactorily, but what the cause of the difference was.

17

As to Maeglin’s greater point, yes this thing does is tell us how 968 people answered a survey.

I agree that we cannot necessarily draw any conclusions about the population at large, nor whether such delusion about these facts genuinely can be correlated with voting for a particular candidate. I would also point out that these particular facts might well have had an enormous effect on this particular election, such that we cannot realistically extend the study any further (ie. those same Bush supporters might now admit the errancy of those statements, and were holding them to be true solely to assuage cognitive dissonance until after the election).

But I would argue that the “eighth grade analysis” shows such a clear discrepancy that these results simply cannot be dismissed as “lacking scientific credibility”. One might argue that it’s not quite as simple as “Bush won because his supporters were deluded”, but one cannot simply deny these IMO shocking results: they must at least be intellectually accomodated.

18

Isn’t this what all polls do? Let’s just not take any more polls then, since all they tell us is what *those * X people think, not what the general population thinks.

This reminds me of a story:

While, strictly speaking, the third logician is correct, in practice, we can easily see that what the second logician says is most probably true.

This is true of the PIPA results. While, strictly speaking, all they tell us are how these 968 people think, in practice, we can infer much more, as SentientMeat’s excellent post about rejection of the null hypothesis shows.

19

I have no idea what you are going on about here. It is basic statistics that as long as a sample is chosen randomly without bias then you can statistically infer about the larger group from the results of the sample. And, when your sample size is on the order of 1000 people then you actually get quite good statistics on the larger group, i.e., what you conclude is good to ± a few percent.

If this were not true, then polls would be useless. And, while there were some problems with polls in this election, that is because the differences in what the two candidates polled was often so small that it was within the margin of error of the poll. However, many if not most of the results from the PIPA poll are so huge effects that they are way, way outside the margin of error of the poll. They could have polled 100 people and had good enough statistics to draw conclusions with a high degree of confidence.

One could posit that there were certain biases in choosing the sample. (E.g, with the regular election polls this season, there was all this worry about missing voters that only had cell phones and no land line.) However, again the effects seen in this poll are often so large that you would really have to come up with an incredibly dramatic bias in order to be able to discount them.

I agree that some of what PIPA talks about in terms of why Bush voters cling to incorrect beliefs and so forth is more in the realm of hypotheses that would need further testing. However, the basic poll results are there for anyone to see and claiming that somehow they tell us only about that particular sample and not about voters as a whole seems to be a statement that is just bizarre from someone who claims to be quite knowledgeable in statistics.

[By the way, we have come across an argument before here on the SDMB that if you have a very large population that you want to learn about then you have to poll some reasonable fraction of that population. This is in fact ignorance of statistics of the most pernicious sort. In fact, as long as your sample is truly a random sampling of the larger group, you can get good statistical results even if your sample size is a very small fraction of the population, which is why polls work at all. In the limit of a large population size, the margin of error is dependent on the sample size alone…The size of that population does not come into at all. And, basically, the margin of error is roughly sqrt(N) where N is the sample size, so if you want an answer good to about 1%, you need N=10000, and N = 1000 gives you roughly a 3% margin of error.]

20

I have to fourth or fifth the criticisms of Maeglin’s critique. Unless you have criticisms of the derivation of the sample, there is nothing wrong with making inferences about the population as a whole. Are you trying to say that they did not conduct any statistical test of the differences? (I’m not that familiar with the study in question). If that’s true, then you have a point.

Are you suggesting that a sample of 968 is small for this purpose? If so, why? How shall we revise the sciences in general, since a sample of this size is in actuality relatively large?

I’m quite experienced in statistical analyses in the social sciences. I am honestly curious about your complaint, and look forward to your elaboration.