Andrew Sulllivan was on a brief vacation, but he comes our roaring–and pretty much nails the situation perfectly, I think:
A statistics lesson, or more likely a cure for insomnia:
Preliminaries:
The population parameter is the actual number you’re trying to estimate with statistics. The sample statistic is that estimate.
The 95% MOE is the radius of the interval around the sample statistic that has 95% likelihood of containing the population parameter. You can use other % MOEs, but we’ll assume 95%.
It’s technically the other way around - that the statistical estimate has 95% chance of being within one MOE of the population parameter. But it doesn’t hurt to think of it in the more natural way.
Now, the fun part:
The thing is, the MOE only applies to each individual statistic, with respect to each corresponding population parameter. Taking the latest SUSA Texas survey as an example, the poll’s sample shows that Obama’s got 49% support, +/- 3.8% MOE, and Hillary’s got 45%, +/- 3.8% MOE.
That means Obama’s actual support has 95% chance of being between 45.2% and 52.8%, and Hillary’s actual support has 95% chance of being between 41.2% and 48.8%.
There’s obviously an overlap there. Hillary’s support could be bigger than Obama’s. The question is, how (un)likely is that? It’s a tad complicated.
If two sample statistics S1 and S2 with margins of error MOE1 and MOE2 are independent - that is, if one is subject to sampling error in one direction, it doesn’t affect what happens to the other one - then the formula for MOE of their difference is:
MOE(S1 - S2) = sqrt(MOE1^2 + MOE2^2).
Example: This Texas survey had Hillary’s support at 45%, +/- 3.8% MOE. The last one had it at 50%, +/- 3.9% MOE. Those are independent because sampling error in one survey that moves Hillary’s apparent support up or down doesn’t affect sampling error in the other one.
So the MOE of the difference between Hillary’s polled support a week ago and Hillary’s polled support now is sqrt[(.038)^2 + (.039)^2] = .054, or 5.4%. So Hillary’s apparent 5% drop in support is within the MOE of the difference. (Ditto Obama’s even smaller apparent 4% increase.)
Now, the genuinely UGLY part:
It gets a LOT hairier when we compare statistics that clearly aren’t independent of one another. For instance, when we compare the support for rival candidates. If we oversample the supporters of one candidate, we’ve almost surely undersampled the supporters of the other, especially when the undecideds are down around 5%.
When two statistics tend to rise or fall together, we say they’re positively correlated. If they tend to move in opposite directions, we say they’re negatively correlated. We measure correlation with a correlation coefficient, r. -1 =< r =< 1, with 1 being perfect positive correlation (moving up or down in lockstep), and -1 being perfect negative correlation (one always goes down if the other goes up, and vice versa).
The support of two candidates running for the same is negatively correlated. How strong that correlation is, depends on the percentage of voters who are either undecided or support other candidates. Right now, if you oversample one candidate, you’re much more likely to be undersampling the other candidate than to be undersampling undecideds, so you’ve got a correlation coefficient near -1.
This is important, because here’s the complicated version of the formula:
MOE(S1 - S2) = sqrt(MOE1^2 + MOE2^2 - 2rMOE1MOE2).
So if Obama’s ahead of Clinton, 49-45, in this latest poll with 3.8% MOE, what’s the MOE of the difference?
It’s sqrt[(.038)^2 + (.038)^2 -2r(.038)(.038)]. I guess we’d better estimate r.
The bad news is that even professional statisticians have to use pretty crude tools to estimate r in these situations. My favorite quickie formula is this:
|r| ~ Average (of the two you’re comparing)/(Average + Other/Undecided).
Basically, that’s the chance that if we oversample one candidate by 1 person, we undersampled the other candidate, rather than Other/Undecided, by 1 person.
Here, the average of Hillary’s and Obama’s support is 47%, and there’s 6% other/undecided. So |r| ~ 47/(47+6) = .887. But the correlation is negative, so r ~ -.887.
So our estimate of the MOE of the difference is
sqrt[(.038)^2 + (.038)^2 -2(-.887)(.038)(.038)]
= sqrt[3.774 * (.038)^2]
= .038*sqrt(3.774)
= .074, or 7.4%.
So Obama’s apparent lead is well within the MOE.
Why SUSA says it’s at the edge of the MOE, I don’t know. The formula above comes from Census Bureau Source and Accuracy statements such as the one in Appendix G of this ugly mofo of a PDF. (See formula (3) on p. G-11 in particular. The calculation for MOE of a difference is the same as for a standard error of a difference, just with everything multiplied by 1.96.) And it’s not like the rules of statistics work differently for different organizations. Nor is SUSA’s original MOE that of a difference; given a sample size of 704, 3.8% is the right MOE for the estimates themselves.
Was that far more than you wanted to know, or what?
I think Spoon said (sang) it better:
Thanks to the Clintons, I’ve had that song going through my head for weeks.
EEEEEEEEEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
You’re hurting my head here, people!
Please make it stop.
Your points are well taken, you with the face, so let me step back and make sure I am articulating myself here.
I think the biggest distinction between interpreting the CI statistics in two way polls and in other areas of science is that a poll can still contain useful information even if, from the perspective of hypothesis testing, you fail to find significance. One candidate has to be leading another, so it is possible to use tests to determine this. Hence the MoE should not be overinterpreted.
But, on edit, RT has waded in. I am a small s statistician, he is the real thing. So I rest my case. 
On edit, a little googling revealed something called “Confidence Limit Misconception.” So a 95% confidence interval does not mean that 95% of the replication actually falls into this interval. Go figure.
Sorry to come in on the end of this statistics discussion without referencing any of the prior discussion. I’d been composing my post hours ago, then had to break to do some work. To borrow the Bush Administration terminology, ‘no one could have foreseen’ that Richard Parker’s quick question would have fomented a serious stats discussion!
I’ll take some of the blame. There is a 95% probability that when I leave the office today, I will be shot.
I should be breaking to do some work, but my current model is not working, and I need to convince myself that I at least have a vague idea of what I am doing.
Is it just happenstance that 7.4 is pretty darn close to 3.8 *2, or 7.6?
I’m not going to disagree with you on this. It just seemed like you were going to far opposite extreme by was calling it irrelevant.
I’m not a hardcore statistician myself, but I think it’s fair to say that most of the time MoE can give us idea of how firm we can be with our assessments. An apparent two-point lead coupled with a 4-point MoE may tell you a little something or another probablistically, but not with any confidence that you take to the bank. And I think that’s what matters most.
Ok, I think we agree, mostly at least. I just fulminate against the idea of the “statistical tie”, reported when the difference falls within the MoE of the point estimates.
I’d dial back the rhetoric and say that the point estimate MoE is only “irrelevant” in a fairly limited and technical sense, but in practice, it is a reasonable heuristic indicator of how close things really are.
Ahem. Back to the fork in progress. . .
Two more superdelegates have endorsed Barack Obama today. Shadow United States Senators Paul Strauss and Michael D. Brown. . .
I think you can draw some fairly damning parallels between the Clinton campaign and the Bush administration. It sure looks like she rewards loyalty over competence, that she didn’t plan for the long term or even the possibility of the long term, that the benchmarks of what is and isn’t important keep changing (I love the phrase, whoever coined it, of “the insult 40 states strategy”), and that somehow it’s never her fault. Heck, I’m surprised there haven’t been more opinon pieces along those lines.
Not at all, but I’m headed out the door. I’ll explain later, if nobody beats me to it!
Well, sort of – it’s due to the low number of undecideds in the sample (which I guess means it’s not happenstance)
If you look at the formulas that RT provided (and, by the way, it’s posts like that that make me love the SDMB!), you can quickly see that if you’re comparing two samples with the same MOE (call it m) where the samples are independent (r = 0), the MOE of the difference is going to be (1.4 * m) and when the samples are completely negatively correlated (r = -1), the MOE of the difference is going to be (2 * m). So the closer r gets to -1, the closer the MOE of the difference will be to (2 * m).
Because there are so few undecideds in the survey, it’s not surprising that r is pretty closer to -1, and thus it’s not surprising that 7.4 is pretty close to 2 * 3.8.
(Something interesting I just realized, but which makes perfect sense upon reflection, is that if r = 1, the MOE of the difference is 0)
That’s far more than I’m capable of knowing.
The Bright Side:
We obviously have to look at the bright side of all of this. Hillary’s stupid and selfish win at all costs cloud will have a sliver lining. We will be finally rid of DLC goons like Mark Penn and Terry MacAullife and the rest of the people aligned with them. Notice that this is what people at places like DailyKos are essentially set up to do. It’s set up for more Democrats and better democrats. It’s to give primary challenges to those pathetic Democrats who are essentially corporate shills.
We’ll finally be exorcised of the Clintons in the Democratic party. Her erratic behavior of late is essentially the end of her political career. Sure she can finish serving her term as a Senator, but if she goes much further in her “nuclear” strategy, then she ought to leave after that.
Another aspect is that the long challenge has forced Obama to set up extensive organizations in so many freaking states that it’s mind-blowing. I wonder what this will do for us in the general election? I think it will mean a landslide victory for Obama. He’s organized in nearly every single state. I can’t imagine why that wouldn’t help out tremendously in the general.
The bright side is that we’ve really dodged a bullet in Hillary. She won’t win if nominated (obviously won’t happen). But the other thing is that if somehow McCain beats Obama she won’t win the next round either. She’s simply gone too far in her quest. It has shown us who the Clintons truely are.
Hillary has repeatedly shit on the process all the way to the nomination. In the beginning she shat all over it by saying categorically that she “would be the nominee.” This is a tactic that isn’t new, yet from her huge advantages already it was clear that she had little to stop her. As soon as it became clear that she might not win without a super-delegate coup she decided that voting didn’t matter. The American people have just gotten tired of being shat on by Hillary. She has shown zero decency and humilty throughout the process. She’s done nothing but pump up pathetic wins and make excuses for her huge losses. Did you hear her excuse about Wisconsin? “We spent less money and didn’t have as much organization.” How is that supposed to make people feel better? Seriously. That’s WHY people lose elections. It’s like saying, “we did quite well despite the fact that more people like Barack”
This will be good for the general, because McCain probably won’t hit on Obama all that hard. I’m sure the Republicans will, but McCain himself most likely won’t. Hillary will be thoroughly discredited and her legacy and Bill’s will be a joke.
Hillary is the embodiment of what is wrong with the Democratic party, and also what is wrong with American politics in general. When she dies, these things will die with it. The Democratic party will move into a new era with Howard Dean (Mr. 50 state strategy himself) and Barack Obama as spiritual leaders. But it really required a complete and thoroughly public trashing of this kind of politics for the other thing to come about. I was really hoping to see a similar situation with the Republican party too, but they didn’t really get the picture this time around. Maybe we’ll see it in four years when some anti-Bush republican runs for President.
MilTan beat me to it, and did a great job, too. I can’t think of a thing to add.
Thanks guys! It’s amazing how much more “r” means when it actually, well, means something, than when it’s problem 9.17 that’s due on Thursday. Makes perfect sense now.
While there are a few DLC Dems who didn’t sign up for Team Hillary, seeing the backs of those that did will be a Good Thing indeed.
I’ve been wondering, though, why there wasn’t more of that this year than there was. After sticking Lieberman with a primary loss in 2006, I figured the netroots would be backing a dozen or so primary challengers this time - and I was prepared to open my wallet. Instead, there were only two: Mark Pera challenging Dan Lipinski, and Donna Edwards taking on and defeating Al Wynn.
Sure, we won 1 out of 2, but 1 out of 10 or 12 would have sent a stronger message.
This is one reason why I finally tilted towards Obama after Edwards dropped out. The level of organization he’s created is going to be a game-changer in the general election. No national Democratic candidate has had anything like this in my lifetime. And I really think it may put some House and Senate candidates over the top in red states that Obama himself doesn’t win in, just by organizing and getting so many extra Dems to the polls. If, for instance, Rick Noriega wins his Senate race in Texas, this will be a big part of the reason why.
This I believe is the single largest reason Obama is the dem to beat in the general. As long as Mar.4 goes his way - and I can concieve of a sweep again - I see no reason why he wouldn’t obliterate the repub candidate in November