Survey Response Error: Variance v. Bias (Another Election Thread)

Great Debates
1

When a statistical survey is taken, there are several steps at which error can occur. Most of them have to do with whether those people actually surveyed are a representative sample of the population you wanted to gather information about. Fortunately, none of that applies to an election (and, yes, this is about that): you assume that the people who cared to get to the polling place and vote are the people who are supposed to decide the issue.

However, once you get a ballot into the voters’ hands, there are still two types of potential error worth discussing: variance and bias. Mind if I use an analogy?

Suppose you weigh 175 pounds. You step on the bathroom scale a bunch of times; it gives you a range of numbers between 170 and 180. That’s variance: the scale isn’t giving you an exactly correct reading, but any error you get is just as likely to be in one direction as in the other.

Now suppose your scale instead consistently reads 168 when you step on it. That’s bias: it’s giving you a reading that’s consistently skewed to one side.

Statisticians are willing to live with variance; a certain amount of it is inevitable. We like to try to keep it small where we can, but variance, like the poor, will always be with us. But bias, OTOH, is just plain bad: it gives you false results.

Now, let’s talk about Palm Beach.

People have been saying that there are always going to be voting errors; not every vote will be recorded correctly. As a Census Bureau employee, I have a fairly good grasp on the problems involved in counting every American correctly; I not only sympathize with the voting officials when they say that, but agree 100% that that’s the case. There’s no way to get rid of noise and random error in the system, no matter how much money you spend to make a perfect process.

The problem with the Palm Beach ballot is that every bit of evidence suggests that the ballot introduced bias into the result. If what you’re trying to measure is, “How many more Palm Beach voters intended to vote for Gore than for Bush?”, then it’s hard to argue that this ballot was as likely to overstate that number as it was to understate it.

First, there’s the evidence of our eyes: you look at the ballot, and it’s pretty clear that if anyone’s going to be confused about where to punch that card, it’s going to be a voter who’s trying to vote for one of the names somewhere in the middle of the ballot (such as Gore, or Browne, or Nader); if voting for a candidate at the very top (such as Bush) or very bottom, it’s pretty hard to get it wrong.

Second, there’s the anecdotal evidence. Many people have come forward, saying they thought they voted for Gore, but now believe they accidentally voted for Buchanan. If any Bush voters were in the same quandary, they’ve been pretty quiet.

Third, there’s Buchanan’s extremely high PBC vote total - 3.5 times as many votes in PBC as in any other county.

Now if the bias is small, it should be ignored. If it caused a shift of 20 votes out of the Gore column, then who cares? But the possibility exists that the ballot may have shifted thousands of votes out of the Gore column, turning them into either Buchanan votes or into double-counts.

There’s ways of measuring the size of the bias. The best way would be to use the same structure of ballot for something completely unrelated to the election, like which candy bar the persons surveyed prefer. They can punch the ballot, then they can be asked which candy bar they chose. If the numbers are skewed in ways that correspond with poition on the ballot, then you know you’ve got a biased ballot, and you can measure the approximate strength of the bias.

Or you can look at the double-counts. There are 19,120 ballots which were disqualified in the Presidential voting for double-punching. Given the unlikelihood that more than a few people were in the voting booths, agonizing over whether to vote for Gore or Buchanan, we can assume that just about all Gore-Buchanan double-punches were accidental. Tossing out any ballots with three or more holes punched, we can total up how many people double-punched (a) Bush and Buchanan; (b) Gore and Buchanan; © Bush and anyone besides Gore and Buchanan; (d) Gore and anyone besides Bush and Buchanan.

If (b) sticks out like a sore thumb among the totals, then it’s hard to see how anyone could argue that Gore wasn’t losing thousands of votes to the confusion over which hole to punch. If as many double-punches involve Bush as involve Gore, then you’d have to conclude that RT’s been making a big deal out of nothing here. And you could get a lot of results in between.

But I don’t think Palm Beach County should be certified until those 19,120 disqualified ballots are tallied to see just who the double-punches involve. You can put up with a lot of variance, and even a little bias. But if you’ve got a ballot that is inducing large amounts of bias into the election results, then that’s a different story.

2

Now, can we see to it that this information, presented exactly this way, get to the people who can make this decision?

stoid

3

Speaking of bias, wouldn’t the positions of the names on the ballot invite a certain amount of bias in themselves? I’m sure it isn’t as significant in the big elections as it is in, say, school board races with a whole bunch of candidates, where a lot of people don’t know them and might just pick the first one. Electronic voting systems could correct this bias by rotating the names.

Since variance and bias are unavoidable, I think that any election that falls within a given margin (300 in 6 million certainly being acceptable) should be called a tie. In most cases that leaves the question of what to do with the result, but the solution here is simple–split the electoral votes. (No, I don’t know who should get the extra one–give it to Nader. :slight_smile: ) Whether it would push Bush or Gore over the top (a still-unanswered question, pending recounts in other states), I think that would be the fairest way of going about it.

Dr. J

4

and split the electorial votes to 7 decimal places:)

5

Doctor J: The order in which the names were supposed to appear were decided by the results in the last Florida governor’s election. So Republican should have been first (it was) and Democratic should have been second (it was in the reading, but not in the actual hole).

6

Stoidela - thanks for the compliment! In answer to your question, I’m trying. Skip down a few paragraphs, if you will.

DoctorJ - They’ve done research on the effect of positioning in a list, and I wish I were more familiar with it. I know it’s big in telephone surveys once the list of choices gets beyond 5 items (after that, the last and first choices seem to do better), but since I haven’t heard much about the effect of order per se on a printed questionnaire, my suspicion is that they’ve found it has little effect.

The courts have ruled that ballot order, by itself, is not something that a candidate can sue about. Since, on paper ballots or mechanical voting machines, somebody’s got to be first, I can’t really quarrel with them too much on that. And in a race where people are genuinely aware of who’s who (like this one), my guess is that the statistical effect of ballot order is pretty close to zero.

My brother-in-law used to be a veep in a political activist group that looks like they’re gearing up to file an amicus brief in any suit over the Palm Beach ballots. He still consults for them, so he’s still well-connected over there. Since Thursday, when the thought first crossed my mind, I’ve been bugging the hell out of him with emails, asking him to pass the message through that they need, as part of their team, a survey statistician who does research in questionnaire design, and explaining just why they need one. I’m not such an expert - I’m really a mathematician who knows just enough statistics to get by - but I’d know who to talk to if I wanted to find one. He says he’s passed my name and arguments along to his old outfit, but that was just going into the weekend. If I hear anything at all, it won’t be until Monday.

I also sent an email to the DNC, along the same lines, but the email equivalent of a cold call doesn’t usually do much. Hence the rattling of my BIL’s cage. :slight_smile: If anything comes of my activity, I’ll let you know.

7

RTFirefly: You’re missing an important point - there are no doubt other instances of BIAS in this election. Some that favor Gore, and some that favor Bush. Palm beach is an arbitrary selection. In this case, the bias happens to favor Bush.

Now, Gore wants a hand recount of three heavily democratic counties. This is guaranteed to return a different result, and is also guaranteed to favor Gore. This introduces INTENTIONAL bias in one select area. And Bush has no recourse, because most of the Republican precincts have certified their results, which means there cannot be a hand count.

As a statistician you must understand the danger of selectively screening data, especially data that apparently comes from a biased sample set.

I’d have no problem at all with a hand-count of the entire state, or even a process that says that an equal representation of Bush districts must also be counted (i.e. if there are 2 million votes in Broward county, which went 70% for Gore, then if you want a hand-count there you must hand-count a similar population with a similar skew towards Bush, or a numerical equivalent (i.e. if an area only went 60% for Bush, it should be twice the size).

If you did that, you could eliminate the original bias (which you’re right about), without introducing a new bias in the opposite direction. Which is what is happening now.

8

What if you do establish bias related to the butterfly ballot? Do we conduct a similar exercise in every county in the United States, just to be sure they’re OK too? Or do we stick to the long-established notion that–except for exceptions as egregious as fraud–these types of election flaws are best resolved prior to the actual election.

If you state that the bias won’t be shown, of course, until after the votes are tallied, again, I’d ask if you think we ought to subject every U.S. county’s results to a statistician’s analysis to make sure there is nothing in the data to suggest bias or some other statistical anomoly. Because–to take Sam Stone’s point even further–if you do this only in Palm Beach, or even across the entire state of Florida, you have by design installed a bias in the process relative to the rest of the country. We will then accept only results that withstand this scrutiny, I guess, that show a certain statistical precision in how the counties and categories correlate???

I believe there was confusion in the voting. As I stated in another thread, those confused individuals have rights that deserve protection (and part of that duty is incumbent upon them). The time when that protection ought to have taken place was prior to the election, when the ballot was filed and made available to every registered voter in the county.

9

As I understand it- admittedly based on incomplete knowledge- there is a bias in certain counties, unrelated to the issue of butterfly ballots. Rather it is that punchcard ballots are used in only a few counties, which coincidently are mostly heavily Democratic counties. The punchcard tallies are prone to undercounts since if the chad is not completely separated from the card the machines will read it as a nonvote. That is why handcounts may change the count significantly in these counties. Humans can see what the voter intended when a chad is hanging from the edge of a punchhole, while machines cannot.

I do not have definite knowledge of which counties use punchcard ballots, but if what I have heard, that they are mostly Democratic, is accurate, then there is certainly an anti-Gore bias in machine tallies. Further, unless someone can propose why there would be an anti-Bush bias of similar magnitude in other counties, it is not arbitary to select Palm Beach and other large-population punchcard counties for manual recounts.

10

I think this ‘bias against Gore’ is a bit of a fabrication, or else the Democrats are morons. Are you suggesting that this obvious wisdom was not available to the Democratic supervisors before the election? If you knew that your district was heavily democratic, wouldn’t you take steps to make sure that the machines counting the votes wouldn’t under-count?

There’s another form of bias inserted here - bias from wear on the ballots from the machines, and from flaws in manufacture. Those ballots have been through the machines twice, and have been heavily manipulated by hand. If, for example, the perforating machine where the ballots were created made the first punch-hole slightly stronger than the rest, then you’ll get a bunch of false positives that bias against Bush. You’ll get some false positives (and some false spoiled ballots) simply from wear and flaws causing chards to fall out.

11

Sam, in responding to your first post, I’ll just point out that, while there may be other instances of bias in the election, Palm Beach is the only visible instance of bias. And it sticks out like a sore thumb.

Regarding the recount, it seems to me that, from the beginning, both candidates were free to choose what to do. My understanding is that Gore’s request for a hand recount was based on discrepancies between the two machine counts (the ones, btw, that the Bush camp are claiming to be so precise). I’d say it’s up to Gore to ask for recounts in places where he believes Democratic votes might’ve gone uncounted, and leave Bush to be the guardian of similar Republican concerns.

At any rate, I don’t give a flying flip about the recount; that’s not my issue. I’m talking about survey error, not policy error, which is what the choice of recounts would fall into.

Bob, if anyone knows of an instance of response bias in Tuesday’s balloting that favored Gore (and the contesting of which would favor Bush), then they should bring it to light. This one didn’t need to be searched for; it was in plain sight almost immediately.

You have to rely, to some modest extent, on the individual-responsibility stuff that I keep hearing about: if someone personally affected by a ballot that was biased toward Gore isn’t capable of calling in to Rush Limbaugh, who I’m sure would be happy to grab that ball and run with it whether it had any air in it or not, then I’m not gonna go looking for that person.

Sam, in answer to your second post, apparently while they’d used the same ballot gizmo before, they’d never before needed to consider putting candidates for one office on both sides. So they hadn’t run into this particular confusion before.

As far as which party was in charge of designing the thing, I’m not proposing fraud; the relevant fact is that those who decided to use that ballot mechanism in this particular manner were laypersons, not survey statisticians, and ‘common sense’ wasn’t enough to keep them out of trouble.

Finally, I don’t understand how putting the ballots through the machines twice creates bias. Would Gore ballots wear faster than Bush ballots? Seems to me that any effect of the wear and tear would be equally likely to affect PBC’s Gore-minus-Bush differential in either direction, which would make it variance, rather than bias.

12

Not necessarily. It might make other forms of bias appear. For instance, if the perforating teeth in the factory were slighly sharper on a particular hole, it might be more likely than others to fall out because of wear.

Or if the counting machines wear the cards unevenly.

Or if the way cards are bent when they are picked up puts more stress on the middle chards than on the edges, or vice versa.

13

In your opinion.

I too have seen the ballot in question. All the voters had to do is follow the arrows. Remember, these are the same people that appear to be perfectly capable of following the arrows on the highways of Florida and arriving at their desired destinations with little trouble. And if they can do this while moving at 60 mph; they certainly should be capable of doing the same while standing still in a voting booth. It’s time to quit whining.

14

It doesn’t matter that the ballot shouldn’t have been confusing. The point is that the ballot was confusing to a small but significant number of people, and that confusion hurt one candidate significantly more than the other. It doesn’t matter that the bias wasn’t intentional, or wasn’t foreseeable–it was still a biased study.

RTF:

True for voting booths like we have here in KY. With paper ballots, though, you could very easily print up several different ballots that rotate the names around. If that had happened in Florida, the same number of people might have screwed up their ballots, but it would have affected all the candidates proportionally, eliminating that particular source of bias.

I would agree that it is close to zero, but I think it could easily account for 300 votes in 6 million (0.005%). That’s close enough for me.

Dr. J

15

And in the three types of evidence I cited in the OP, Unc. At this point, it’s the Bushies who have nothing but opinion on their side.

Mabe is should’ve been a snap to follow the ballot. But if you put any burden on the voters for one side, but not for the other - no matter how trivial - it’s going to affect the outcome. If we’d required every Bush voter in Palm Beach County to spell ‘potato’ correctly to have their vote counted (a task that any 7 year old should be able to handle), that would’ve probably knocked out a couple thousand Bush votes, and Gore would be the President-elect.

And the reverse of that is what happened to Gore in PBC.

Anyone here remember what Hari Seldon said?

16

“Maybe it should’ve been a snap…”

And my apologies about the bolding. Preview is my friend… :wink:

17

All right, RTF, let’s deal with your evidence.

  1. “First, there’s the evidence of our eyes:”
    Your eyes maybe, not mine. I can clearly see which arrows point to which punch holes. This is a subjective judgement on your part. I draw different conclusions given the same data set. Anyway, all parties, prior to the election approved the ballot. Whining about an inherent unfairness of the ballot design after the fact, is disingenuous.

  2. Second, there’s the anecdotal evidence"
    Anecdotal evidence is a pretty poor basis for attempting to overturn a vote count. You, and the rest of the Teeming Millions (me included) would not accept “anecdotal evidence” in support of something so simple as an opinion stated in this forum. I’d like something a bit more concrete than anecdotes. I saw someone vocalizing one of these anecdotes on TV yesterday morning. He said, “I’m sure I voted for Gore, but now I think I may have punched my ballot for Buchanan.” It was clear he really had no real idea which hole he’d punched. Ask David B about “false memories.” Hell, se had someone here swear he’d seen the “Up the Butt” episode of The Newlywed Game. Again, anecdotes are very flimsy rationale on which to attempt to overturn an election. You’re gonna need more than that to show bias.

  3. “Third, there’s Buchanan’s extremely high PBC vote total:”
    So? This does not prove bias in and of itself. If your other points hold this might be evidence of a bias, but I don’t believe they hold water, but it might also be evidence of people actually intending to vote for Buchanan. So, I’m dismissing this also.

So, again, bias is shown only in your subjective opinion.

I’d just like to address one more point you made in your last post, RTFirefly. " If we’d required every Bush voter in Palm Beach County to spell ‘potato’ correctly to have their vote counted (a task that any 7 year old should be able to handle), that would’ve probably knocked out a couple thousand Bush votes, and Gore would be the President-elect."

Ya got anything but anecdotes to support this? This is an inflammatory statement and has no place in a rational debate. It certainly does nothing to support your cause and you should be ashamed of posting it.

18

Well said, Unclebeer. Have you checked out the “Why I don’t trust Republicans/conservatives” thread?

19

I’ll get there eventually, Scylla, but I’m trying to meter my production of stomach acid today. I’m out of Rolaids at the moment.

20

UncleBeer, this is Florida we’re talking about. Especially in Palm Beach County, these people don’t drive over 35 mph, regardless of the speed limit or anything else. :wink:

More seriously, I think following highway signs is a bit different than following that ballot. Look, FWIW, I agree with you - the thing seemed pretty clear to me. Then again, I’m able to follow the signs on the New York side of the Holland Tunnel, my eyesight is 20/20 and I did really well in reading comprehension.

That said, while I’d love to see Gore win the election, I think that both candidates should stand pat, cut the campaigning and wait for the absentee and military ballots. After Friday, whoever’s ahead wins. The country cannot afford too long a delay, and all this crap about Gore being named Bush’s VP just ain’t going to happen. A long court fight would merely erode the legitimacy of whoever ended up in the White House.

And while I’d also love to see some voting reforms based on the to-do in Florida, if the country won’t accept statistical sampling for the census I don’t see those in power accepting or acting on other statistical arguments (e.g., the trade-off between making it easier to convict the guilty and convicting a few innocent people, which is either a Type 1 or Type 2 error, I forget which).