When calculating the margin of error in a scientific experiment, you look at how exact your measurements are: ie my ruler only measures to the nearest millimeter.
How is it handled for polls? Let’s say it’s election day, and the race for dog catcher is predicted to be 57% to candidate A and 43% to candidate B, with a margin of error of 6%.
Why 6%? Why not 8% or 4%?
I don’t remember the details of how to do it from my stats classes, but you base the calculation on such things as the size of the sample and the standard deviation of the results. The calculation estimates the likely variation if you took multiple samples from the population, and the margin of error is usually based on the 95% confidence level, i.e., you are 95% confident that the results are within the margin of error.
A bigger sample means a smaller margin of error, but bigger samples cost more.
It’s related to a Confidence Interval - iow, what level of certainty do you have that the emprical data matches the true distribution?
To further expand on Giles explanation, the results you gave mean that we can be 95% (usually) sure that candidate A will receive between 51-65% and candidate B will get between 37-49%. Which means that it’s very, very likely indeed that A will win this one. If, instead, the poll results were more like 49% for A and 43% for B, it’s less likely that our prediction will be correct, because of the degree of overlaps (43-54% against 37-49%).
Plus, of course, there’s a 5% chance that these results aren’t accurate anyway.
Depending on the sample size, there are established mathematical formulas for varying degrees of confidence. I could probably find them in my old stats textbook if anyone rteally wants to see them.
I would like to see them, but don’t rush, I’m not in a hurry for mathematical formulas belive it or not.
I can’t look it up right now, but if you wiki “omega”, “standard deviation”, “confidence interval”, “xi squared table”, and “margin of error”, you’re bound to find enough info to satisfy your curiosity. I learned how to calculate that in 11th grade, so it’s not highly technical. I seem to recall it being especially easy if you know how big the sample population is. For elections, we have a pretty good, if not exact, count of eligible voters. You can probably do the calculation in your head.
In the OP example, both individuals share a 6% MoE. Is this a coincidence, i.e. were both separately calculated to have a similar MoE? Or could they be assumed to be equal if homogeneity of variance is assumed?
It’s the same population and the same sample, so wouldn’t you expect the variance to be the same?
In addition, the results seem to be based on just counting those who say they will vote for one of two candidates: there’s no third candidate, there’s no "Don’t know"s, and there’s no "Won’t vote"s. So if candidate A gets X%, candidate B automatically gets 100-X%, and the variance is automatically the same.
The margins of error reported with polls bug me. Confidence intervals rely on a confidence level, which is never reported. The confidence interval just measures how likely it is that your sample will match the general population.
Let’s say you’re playing blackjack and you know aces generally help the player. If you can see all the cards at once, then you can tell if there are the right number of aces. However, you can’t. You can only see a sample of cards as they go by, and there are shuffles in between hands. So if you see 130 cards, you expect one of them to be an ace, and you might be 90% sure that there will 10 +/- 1 (i.e. 10%). In 1300 cards, you expect 100 aces, and are 99% sure there will 100 +/- 10%. I’m not doing the math here, but generally, the bigger your sample, the smaller your confidence interval for a given confidence level. Without knowing the confidence level, the size of the interval is meaningless.
The +/- 3% also hides other problems. Was the sample made completely randomly? Were the questions misleading? Did the interviewer influence the responses?
Margin of error tries to strike a balance between two extreme methods of interpreting a poll:
(1) Dismissing the results altogether because any result is possible–after all, in a large population where only 1% supports candidate A, there is a miniscule chance that everyone in your random sample happens to support A, so you will report the wildly-wrong answer of 100%.
(2) Assuming the perfect accuracy of any result given. A poll shouldn’t be discredited because it said candidate A had 50% support when he/she really had 50.1%. The question is, how close is “close enough”?
To deal with (1), you assume a “confidence level”, which concedes that yes, this very very unlikely scenario could occur, but suppose I throw out the most egregious 1% of possible mis-readings (i.e. the ones where the polling is the farthest away from the actual value). What can I say about the remaining possible readings?
Surprisingly (if you haven’t studied statistics), there’s a lot you can say, because for even a modest sample this 1% covers a wide range of error values. This in fact allows us to deal with (2) by saying “I don’t guarantee perfect results, but I’m 99% sure that by choosing at least X people at random I’m within Y% of the right value, which is close enough for me.”
As an example, for a poll of 1,000 people (taken from, say, the ~150 million registered votes in the US), the result of that poll is likely to be within 4% of the true value 99% of the time, i.e. if 100 pollsters ran the same poll on the same population (each choosing different 1000-person samples), only one of them is likely to have a value that is more than 4% away from the actual value.
In my experience most pollsters run at a 95% confidence level. For this level of confidence, the margin of error is ~ 1/sqrt(n), where n is the number of people in your sample. Thus, a poll of 100 people would produce a result with a margin of error = 1/sqrt(100) = 1/10 = 10%, i.e. it’s 95% likely the real result is within 10% of the true value. If you want to boost your confidence to 99%, just add another 30% to the margin, e.g. that same poll would have a margin of error = 13% if you wanted to be 99% confident. It’s therefore more valuable to know the number of persons in a polling sample, since with this you can easily calculate either margin of error.
Finally, this value refers to the expected statistical effect assuming a purely random sample, so it is a “best possible” value–the math says you can’t expect the results to be any better, and if a poor methodology is used to select a “random” sample the potential error will be worse. Given the speed with which polling a large group can be done today, this is a far more important factor, which it why it’s worth reviewing a poll’s methodology before accepting its results, despite any veneer of legitimacy a “margin of error” calculation can lend it.
Note the item CJJ mentioned, about how accurate the poll taker was in getting a ‘random’ sample. Because that can be much more important than the statistical margin of error.
The “Dewey defeats Truman” poll of 1948 was wrong because of this – their sample was selected from a group that was more wealthy than average, and thus biased toward the Republican candidate.
In 2008, most polls are done via phone. There was significant worry that they were under-representing people who have no home phone, only a cell phone in their sample. (And those tend to be younger people, thus tended to lean for Obama.) Polls did various things to counteract this.
Local newspapers/media often have inaccurate polls because they select a ‘random’ sample of local residents, rather than of local voters – a significantly different group. Or they rely on residents word about whether the vote – people frequently lie about this.
You need to put a fair amount of work & thought into selecting your ‘random’ sample to have a good (accurate) poll.
Thanks for all the answers.
This is very non-intuitive for me. Having not studied statistics or polling methods, saying that I can talk to 1000 out of 150,000,000 votes and be that sure of my outcome flies in the face of all my physical science training.
If you had 150,000,000 tokens, 60% of them are red and 40% are blue, you then take a random sample of 1000 of them, would you be reasonably confident that you’d get close to a 60/40 split?
These error margins have always bugged me too. They are basically just reporting approximately one over the square root of the sample size. The major flaw in the argument, as others have noted, is that the confidence in the results relies on the assumption of a truly random sample of the underlying population. In practice, this is essentially impossible to obtain. Another problem is that people don’t necessarily answer as they would vote. Neither of these issues is addressed when the newscaster reports that the poll has a margin of error of 3%.
But the fact is that these polls are pretty accurate.
For example, in the Presidential election of 2008, 11 national polls as of 4 days before the election (Oct 30th) were showing Obama as follows:
+3% (Fox), +3, +4.1, +5, +5, +6, +6.9, +7, +7, +8, +13,
for an overall average of +6.18% for Obama. The actual results were +6.8%.
So these polls, using about 1,000 people each, were pretty accurate in predicting the results of 125 million voters.
The margin of error always converges on + or - 3% as the sample size rises about ~1500 or so regardless of the population size (as long as the population size is many times the sample size). There’s a lot of math to prove this point but the most intuitive way I’ve seen is to imagine you’re tasting the salt level of a well stirred pot of soup with a spoon. It doesn’t matter how big the pot of soup is, just how big the spoon is.
Also, the + or - 3% error rate only applies to sampling error and does not apply to other sources of error such as sample bias or poorly designed polling. These other errors quite often swamp sampling error and so you need to look carefully at study design to check if it’s trustworthy or not.
It’s non-intuitive for almost everyone. Nevertheless, it’s true. t’s always amazing how little sample size you need to get a reasonably accurate answer.
But (as others have said), it’s not any sample of 1,000. It has to be an unbiased sample, i.e., one in which every member of the population has an equal chance of being chosen. That can be very hard, especially if you don’t have a complete list of every member of the population.
Well, duh? Are we reinforcing every precept of Stats 101?