# Nate Silver is systematically building a case for declaring pollster Strategic Vision, LLC a fraud **URL:** \ **Category:** Great Debates\ **Created:** [September 27, 2009, 4:36am UTC](https://boards.straightdope.com/t/nate-silver-is-systematically-building-a-case-for-declaring-pollster-strategic-vision-llc-a-fraud/511728 "2009-09-27T04:36:46Z")\ **Posts on this page:** 16\ **Page:** 3
**Author:** ![astro](https://avatars.discourse-cdn.com/v4/letter/a/9dc877/32.png) [@astro](https://boards.straightdope.com/u/astro)\ **Post date:** [October 2, 2009, 3:07pm UTC](https://boards.straightdope.com/t/nate-silver-is-systematically-building-a-case-for-declaring-pollster-strategic-vision-llc-a-fraud/511728/41 "2009-10-02T15:07:49Z")
> [@Fotheringay-Phipps](#): > > So that’s my best guess. That SV is not an outright fraud. But they are not a highly capable pollster either. They don’t want anyone looking too closely at what they are doing because they will be exposed as PR hacks. But they’re not completely making up the polls either. > > I could be wrong. I’m curious as to how it turns out. “Not an outright fraud”? A sketchy pollster that operates out of a series of UPS mailbox addresses and has a high school poll with absurd results that it’s fairly obvious are completely made up. I’m not a statistics maven, but the results of that HS survey are so insane I’d like to know how you can characterize them as anything other than fraud. ---
**Author:** ![Fotheringay-Phipps](https://avatars.discourse-cdn.com/v4/letter/f/d9b06d/32.png) [@Fotheringay-Phipps](https://boards.straightdope.com/u/Fotheringay-Phipps)\ **Post date:** [October 2, 2009, 3:52pm UTC](https://boards.straightdope.com/t/nate-silver-is-systematically-building-a-case-for-declaring-pollster-strategic-vision-llc-a-fraud/511728/42 "2009-10-02T15:52:55Z")
> [@astro](#): > > A sketchy pollster that operates out of a series of UPS mailbox addresses I thought that type of accusation is just noise. It’s fairly common for companies to use UPS boxes as offices, or “virtual” offices, that consist of some outfit hired to answer the phone (for a host of companies using the same service). I’m hardly surprised that a PR firm does this - that’s probably exactly what they’re recommending to half their clients. It has zero bearing on the reliability of their polls. > [@](#): > > and has a high school poll with absurd results that it’s fairly obvious are completely made up. > > I’m not a statistics maven, but the results of that HS survey are so insane I’d like to know how you can characterize them as anything other than fraud. I addressed this above. I suspect that they counted spelling mistakes and the like as incorrect answers. These were not multiple choice questions. That said, bear in mind that other surveys have also shown very little knowledge of civics among HS kids. E.g. this survey of California HS kids which shows that > [@](#): > > one half could not correctly identify the function of the Supreme Court, and 33 percent could not correctly identify even one of California’s two senators from a list of options. Forty-one percent didn’t know which of the two parties was most conservative. And these are things constantly in the news. The SV results were more egregious, but not so much more egregious that they are obviously completely made up. ---
**Author:** ![Jas09](https://avatars.discourse-cdn.com/v4/letter/j/d07c76/32.png) [@Jas09](https://boards.straightdope.com/u/Jas09)\ **Post date:** [October 2, 2009, 4:53pm UTC](https://boards.straightdope.com/t/nate-silver-is-systematically-building-a-case-for-declaring-pollster-strategic-vision-llc-a-fraud/511728/43 "2009-10-02T16:53:08Z")
> [@Fotheringay-Phipps](#): > > I addressed this above. I suspect that they counted spelling mistakes and the like as incorrect answers. These were not multiple choice questions. This doesn’t seem to be the case. In the full results the totals seem to add up to 100. For example, the totals for the 2 political parties question were 43% Democrat and Republican, 11% Communist and Republican, 46% Don’t Know. Unless they counted everything but the first two as “Don’t Know” then it had to be a multiple choice (and why those were the choices will be left as an exercise for the reader). Similarly for the Supreme Court justices question - _nobody_ guessed 7 or 11, but all other choices from 5-12 were represented? Seems unlikely unless it was multiple choice. Finally, notice that in your California poll the students “averaged a little over 60% correct on the commonly used survey items”. The average in the OK poll was 27.2%. So, either California students are more than three times as smart as OK students (I’d believe twice as smart…) or there is something dramatically wrong with this SV poll. ---
**Author:** ![Fotheringay-Phipps](https://avatars.discourse-cdn.com/v4/letter/f/d9b06d/32.png) [@Fotheringay-Phipps](https://boards.straightdope.com/u/Fotheringay-Phipps)\ **Post date:** [October 2, 2009, 5:36pm UTC](https://boards.straightdope.com/t/nate-silver-is-systematically-building-a-case-for-declaring-pollster-strategic-vision-llc-a-fraud/511728/44 "2009-10-02T17:36:34Z")
> [@Jas09](#): > > This doesn’t seem to be the case. In the full results the totals seem to add up to 100. For example, the totals for the 2 political parties question were 43% Democrat and Republican, 11% Communist and Republican, 46% Don’t Know. Unless they counted everything but the first two as “Don’t Know” then it had to be a multiple choice (and why those were the choices will be left as an exercise for the reader). > > Similarly for the Supreme Court justices question - _nobody_ guessed 7 or 11, but all other choices from 5-12 were represented? Seems unlikely unless it was multiple choice. I assume that they showed the categories that had most responses and included the others with “don’t know”. The descriptions of the survey say the questions are taken from the USCIS citizenship test, and the results are compared to the results of that test. AFAIK the USCIS citizenship test is not multiple choice. ETA: I see the OCPA article [explicitly says it was not MC](http://www.ocpathink.org/publications/perspective-archives/september-2009-volume-16-number-9/?module=perspective&id=2321) > [@](#): > > In considering the profoundly awful results of this survey, it is important to bear in mind that an open-answer format represents a much higher standard than a multiple-choice-format exam, even with high-quality exams such as the NAEP. After all, in a multiple-choice exam, the correct answer is sitting right in front of you. > > For example, one of the questions in this survey is “Who was the first president of the United States?” If students are given four answers, and one of them is George Washington, they have a 25 percent chance of getting the correct answer even if they have no idea who the first president was. But in an open-answer format, you either know the correct answer or you don’t. It is a higher standard, and it’s the standard applied to applicants for U.S. citizenship. > [@Jas09](#): > > Finally, notice that in your California poll the students “averaged a little over 60% correct on the commonly used survey items”. The average in the OK poll was 27.2%. So, either California students are more than three times as smart as OK students (I’d believe twice as smart…) or there is something dramatically wrong with this SV poll. The survey items are not directly comparable. I would say the California test is a lot easier, requiring name recognition and basic understanding versus recognition of terms like “executive branch” and relatively irrelevant things like length of senate terms or number of justices in the OK one. [BTW, while I would guess that your civic knowledge exceeds both CA & OK students, I’m not so sure of your math … :)] ---
**Author:** ![zut](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/zut/32/2875_2.png) [@zut](https://boards.straightdope.com/u/zut)\ **Post date:** [October 2, 2009, 5:57pm UTC](https://boards.straightdope.com/t/nate-silver-is-systematically-building-a-case-for-declaring-pollster-strategic-vision-llc-a-fraud/511728/45 "2009-10-02T17:57:11Z")
> [@Fotheringay-Phipps](#): > > …I suspect that they counted spelling mistakes and the like as incorrect answers. These were not multiple choice questions. > > That said, bear in mind that other surveys have also shown very little knowledge of civics among HS kids. E.g. this survey of California HS kids which shows that […] And these are things constantly in the news. The SV results were more egregious, but not so much more egregious that they are obviously completely made up. They are obviously completely made up (or at least grossly misrepresented) if you look at the raw answers. As you said, these are _not_ multiple choice questions, and they were asked of 1000 students. Keep that in mind and take a look at [this report](http://www.ocpathink.org/publications/perspective-archives/september-2009-volume-16-number-9/?module=perspective&id=2321). 1. On the question, “Who wrote the Declaration of Independence?” seventy people said “Barack Obama” and twenty said, “Michael Jackson.” Would twenty different students independently pick “Michael Jackson” as a joke answer on _an open-ended non-MC test?_ That doesn’t pass the laugh test. 2. On the question, “We elect a US Senator for how many years?” 34% answered “don’t know” and 11% gave the correct answer of six years. The remaining students answered either ten, four, or two years. Not one of these students answered an odd number, and not one of then gave a crazy joke answer. What are the chances of that? 3. On the question, “Who was the first president of the United States?” fully 140 students answered either “Barack Obama,” “George W. Bush,” or “Richard Nixon.” Other students gave presidential answers; oddly, none answered “Michael Jackson.” Again, doesn’t pass the smell test. 4. The kicker: On the question “What are the two major political parties in the United States?” there were only three answers given: “Democrat & Republican,” “Don’t know,” and “Communist & Republican,” the last by fully 110 students. Does it seem plausible that, on a _non-multiple-choice_ test, 11% of the respondents got exactly the same wrong answer, and _all_ other students either got the right answer or admited not knowing? I’m surprised anyone with access to the raw data would think this is a legitimate poll. ---
**Author:** ![zut](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/zut/32/2875_2.png) [@zut](https://boards.straightdope.com/u/zut)\ **Post date:** [October 2, 2009, 6:14pm UTC](https://boards.straightdope.com/t/nate-silver-is-systematically-building-a-case-for-declaring-pollster-strategic-vision-llc-a-fraud/511728/46 "2009-10-02T18:14:17Z")
> [@Fotheringay-Phipps](#): > > I assume that they showed the categories that had most responses and included the others with “don’t know”. A dangerous assumption. From the previous link, > [@](#): > > In Oklahoma, the telephone surveyors called a sample of 1,000 public high-school students and read the following statement: “On the next 10 questions, I will be asking you questions about American government and history. Give me your best answer, and it is permissible to respond ‘I don’t know.'” “I don’t know” is a very specificly defined answer. Lumping _other_ answers in with it would be misleading at best. Interestingly, [the Arizona test](http://regulus2.azstarnet.com/pdf/pdfs/560.pdf) commissioned by the Goldwater Institute and performed by SV is reported by the Goldwater Institute using (for some questions) an “other” category. **Jas09** ’s observation on the Supreme Court question is still apparent, albeit with revised numbers, as is mine on the Senate terms. I wonder, more than a little, what the “other” category represents on the political party question. ---
**Author:** ![Fotheringay-Phipps](https://avatars.discourse-cdn.com/v4/letter/f/d9b06d/32.png) [@Fotheringay-Phipps](https://boards.straightdope.com/u/Fotheringay-Phipps)\ **Post date:** [October 2, 2009, 6:25pm UTC](https://boards.straightdope.com/t/nate-silver-is-systematically-building-a-case-for-declaring-pollster-strategic-vision-llc-a-fraud/511728/47 "2009-10-02T18:25:28Z")
> [@zut](#): > > 1. On the question, “Who wrote the Declaration of Independence?” seventy people said “Barack Obama” and twenty said, “Michael Jackson.” Would twenty different students independently pick “Michael Jackson” as a joke answer on _an open-ended non-MC test?_ That doesn’t pass the laugh test. I don’t assume it was a joke answer. That poll was taken very shortly after MJ’s death, when he was all over the news and being lionized constantly. > [@](#): > > 1. On the question, “We elect a US Senator for how many years?” 34% answered “don’t know” and 11% gave the correct answer of six years. The remaining students answered either ten, four, or two years. Not one of these students answered an odd number, and not one of then gave a crazy joke answer. What are the chances of that? Length of Senate terms doesn’t lend itself to jokes. I’m speculating that some of the smaller answers were grouped with “don’t know”. > [@](#): > > 1. On the question, “Who was the first president of the United States?” fully 140 students answered either “Barack Obama,” “George W. Bush,” or “Richard Nixon.” Other students gave presidential answers; oddly, none answered “Michael Jackson.” Again, doesn’t pass the smell test. I think people knew enough about MJ to know that he hadn’t been a president. But they knew that he wrote things. > [@](#): > > 1. The kicker: On the question “What are the two major political parties in the United States?” there were only three answers given: “Democrat & Republican,” “Don’t know,” and “Communist & Republican,” the last by fully 110 students. Does it seem plausible that, on a _non-multiple-choice_ test, 11% of the respondents got exactly the same wrong answer, and _all_ other students either got the right answer or admited not knowing? It does look weird. But OK is a very conservative state, and I wonder if some people there associate Democrats with Commies and confuse the kids. Prior comments about “don’t know” apply. ---
**Author:** ![Jas09](https://avatars.discourse-cdn.com/v4/letter/j/d07c76/32.png) [@Jas09](https://boards.straightdope.com/u/Jas09)\ **Post date:** [October 2, 2009, 6:27pm UTC](https://boards.straightdope.com/t/nate-silver-is-systematically-building-a-case-for-declaring-pollster-strategic-vision-llc-a-fraud/511728/48 "2009-10-02T18:27:30Z")
> [@Fotheringay-Phipps](#): > > [BTW, while I would guess that your civic knowledge exceeds both CA & OK students, I’m not so sure of your math … :)] Hah! And with a BS and MS in Electrical Engineering (with a minor in math) you’d think I’d avoid those kind of mistakes. :mad: I guess it’s plausible that they lumped everything that didn’t match the top few vote-getters with “Don’t Know” but that seems like a really stupid way of doing it. Especially if “Don’t Know” is a response explicitly given. “Other” seems a much more rational way of doing it. Either way, the results of the OK survey make absolutely no sense. It’s just not plausible to me that not one of 1000 high-school students got even 8 of those questions correct. The only thing that borders on a possible explanation is that the students just made shit up as a laugh. ---
**Author:** ![zut](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/zut/32/2875_2.png) [@zut](https://boards.straightdope.com/u/zut)\ **Post date:** [October 2, 2009, 6:52pm UTC](https://boards.straightdope.com/t/nate-silver-is-systematically-building-a-case-for-declaring-pollster-strategic-vision-llc-a-fraud/511728/49 "2009-10-02T18:52:38Z")
> [@Fotheringay-Phipps](#): > > I don’t assume it was a joke answer. That poll was taken very shortly after MJ’s death, when he was all over the news and being lionized constantly. > > Length of Senate terms doesn’t lend itself to jokes. I’m speculating that some of the smaller answers were grouped with “don’t know”. And I’m fairly certain that’s incorrect, and certainly certain it’s incorrect in the Arizona poll. And “one million” is a perfectly appropriate joke answer. > [@Fotheringay-Phipps](#): > > I think people knew enough about MJ to know that he hadn’t been a president. But they knew that he wrote things. > > It does look weird. But OK is a very conservative state, and I wonder if some people there associate Democrats with Commies and confuse the kids. Prior comments about “don’t know” apply. Just how laughable do the results have to be before you’re willing to think something might not be kosher? “They knew Michael Jackson wrote things”? Really? “Some people in Oklahoma associate Democrats with Commies and confuse the kids?” Really? Enough so that fully eleven percent of the students actually believe that? That’s an extraordinary claim. Do you actually believe that, or are you speculating in a “well, if the stars aligned and maybe these other things happened, then if you cocked your head just right these numbers could have been produced without resorting to outright fraud” kind of way? The reported result that 75% of high school students in Oklahoma _don’t_ know that George Washington was the first president strike me as implausible. Not implausible enough to disregard the whole poll, but enough to take a good hard look at it. And when you take a good hard look at it, a lot of _other_ things look surprising. And implausible. And laughable. At some point you have to say, “yeah, something’s just not right here.” It’s certainly possible this isn’t outright fraud, but I con’t think of a more likely explanation. ---
**Author:** ![Fotheringay-Phipps](https://avatars.discourse-cdn.com/v4/letter/f/d9b06d/32.png) [@Fotheringay-Phipps](https://boards.straightdope.com/u/Fotheringay-Phipps)\ **Post date:** [October 2, 2009, 7:52pm UTC](https://boards.straightdope.com/t/nate-silver-is-systematically-building-a-case-for-declaring-pollster-strategic-vision-llc-a-fraud/511728/50 "2009-10-02T19:52:42Z")
It’s certainly possible that it’s outright fraud, but I still think shoddy is more likely. I know if I was someone who had paid for one of these surveys I would be asking them for backup data around now. ---
**Author:** ![Chronos](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/chronos/32/134_2.png) [@Chronos](https://boards.straightdope.com/u/Chronos)\ **Post date:** [October 2, 2009, 9:35pm UTC](https://boards.straightdope.com/t/nate-silver-is-systematically-building-a-case-for-declaring-pollster-strategic-vision-llc-a-fraud/511728/51 "2009-10-02T21:35:45Z")
> [@](#): > > I addressed this above. I suspect that they counted spelling mistakes and the like as incorrect answers. These were not multiple choice questions. Even if you grant that, you still have to address the fact that, according to their results, all students in Oklahoma are exactly the same. If they were counting off for spelling, or whatever, then the kids who were better spellers would be expected to do better than the kids who can’t spell. But if some groups of students did better than others, then you would have seen a different set of results on that test. ---
**Author:** ![theR](https://avatars.discourse-cdn.com/v4/letter/t/977dab/32.png) [@theR](https://boards.straightdope.com/u/theR)\ **Post date:** [October 2, 2009, 10:04pm UTC](https://boards.straightdope.com/t/nate-silver-is-systematically-building-a-case-for-declaring-pollster-strategic-vision-llc-a-fraud/511728/52 "2009-10-02T22:04:56Z")
> [@Fotheringay-Phipps](#): > > I addressed this above. I suspect that they counted spelling mistakes and the like as incorrect answers. These were not multiple choice questions. They checked for spelling mistakes and counted them as incorrect for a phone survey? That’s a pretty large leap. ---
**Author:** ![Punoqllads](https://avatars.discourse-cdn.com/v4/letter/p/d2c977/32.png) [@Punoqllads](https://boards.straightdope.com/u/Punoqllads)\ **Post date:** [October 2, 2009, 11:05pm UTC](https://boards.straightdope.com/t/nate-silver-is-systematically-building-a-case-for-declaring-pollster-strategic-vision-llc-a-fraud/511728/53 "2009-10-02T23:05:03Z")
There’s one non-fraudulent way to always get percentages that add up to 100%. Ask exactly 100 people. Of course, that results in a 10% margin of error… ---
**Author:** ![Chronos](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/chronos/32/134_2.png) [@Chronos](https://boards.straightdope.com/u/Chronos)\ **Post date:** [October 4, 2009, 4:49am UTC](https://boards.straightdope.com/t/nate-silver-is-systematically-building-a-case-for-declaring-pollster-strategic-vision-llc-a-fraud/511728/54 "2009-10-04T04:49:11Z")
Or 1000 people, if you then report results to a tenth of a percent. ---
**Author:** ![ftg](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/ftg/32/2801_2.png) [@ftg](https://boards.straightdope.com/u/ftg)\ **Post date:** [November 8, 2009, 10:10pm UTC](https://boards.straightdope.com/t/nate-silver-is-systematically-building-a-case-for-declaring-pollster-strategic-vision-llc-a-fraud/511728/55 "2009-11-08T22:10:20Z")
[Here’s](http://www.fivethirtyeight.com/2009/11/real-oklahoma-students-ace-citizenship.html) yet another article questioning SV, this time in regard to the OK student poll. Several other articles at FiveThirtyEight on this since the last post. ---
**Author:** ![RTFirefly](https://avatars.discourse-cdn.com/v4/letter/r/c77e96/32.png) [@RTFirefly](https://boards.straightdope.com/u/RTFirefly)\ **Post date:** [November 9, 2009, 2:39pm UTC](https://boards.straightdope.com/t/nate-silver-is-systematically-building-a-case-for-declaring-pollster-strategic-vision-llc-a-fraud/511728/56 "2009-11-09T14:39:16Z")
What’s _really_ interesting is that, according to Silver, [Strategic Vision has not published any new poll results](http://www.fivethirtyeight.com/2009/11/skipping-09-elections-strategic-vision.html) since he called them out. [Previous page](https://boards.straightdope.com/t/nate-silver-is-systematically-building-a-case-for-declaring-pollster-strategic-vision-llc-a-fraud/511728.md?page=2)