Less ‘takes on’ than ‘forced into conflict with’, I suppose.
Like Strategic Vision last year this time Research2000 is in trouble as analysts not involved with 538 noticed statistical problems with R2K’s results published at DailyKOS and shared them with Nate Silver over at 538. Now KOS has dropped R2K and is planning to sue and 538 is getting cease and desist letters from R2K attorneys. It’s a mess.
But kudos to Silver for not backing down. And kudos to all involved in polling and sampling for taking the industry to the next level: a level defined by self-analysis and evaluation that can force out the pretenders or the dishonest. Or at least reveal them to be dishonest for the world.
I, for one, am not in love with the fact that we live in a world that makes a Nate Silver possible, not because I dislike the man. On the contrary, I admire him greatly. I just find it off-putting that someone, anyone, can predict the outcomes of events with the percentage of accuracy he has many times exhibited. Nate’s as close to a real life Hari Seldon as I’ve ever heard, and I’m just not that thrilled with the reality of being a data point in a math equation.
Is there a simple summary of the goings on out there?
Or is this a fair description:
From what I gather from the handful of stories and postings, DailyKos contracted Research2000 to do some polling for it. Other people notices statistical problems with the poll results (patterns that should have appeared did not), which strongly suggested that rather than actually polling people, Research2000 was making up results. Kos publicly fired Research2000 (giving credence to the allegations), and Nate Silver at 538 has been providing analysis and commentary about the supposed fraud. Research2000 has (or is about to) brought action against Kos and 538 (for various torts?) and is now suggesting the possibility of criminal charges.
What kind of polling results was Research2000 providing? Were they politically-minded results (e.g., happy puppy liberal isn’t Obama great! or look how evil real Americans think Obama is!)? That is, was Research2000 acting as sort of a push-poll rather than just taking the easy way out and fudging the results?
True, as I just looked it up, but a red herring, my friend. Those polls can be had for either side, for any time. And you know that. You disappoint me that you go for that sort of play.
The story here is that the guys involved in the analysis here (which is not Silver at 538) have again identified, just through the output, polling that looks fishy and have managed to place a firm against a wall with nothing but the math.
With any luck at all a few more cases of this (Strategic Vision, R2K, and others) will lead to greater openness in sourcing, questions, methodology and such that leads to better, non-bias polling. At a minimum it will put the firms that specialize in biased polling into greater relief and place those who use them under a spotlight.
Greater transparency and greater disclosure is NEVER a bad thing. That’s my reporter’s soul speaking. But I think I’m right.
If there’s a lesson to be drawn from this example you bring up, it’s that Markos Moulitsas doesn’t want a pollster who’s simply going to tell him good stuff about his side and bad stuff about the other side - that his career of liberal political activism doesn’t override his desire for the truth.
It’s all the polls they’ve done. In fact, that is the core data that exposed the (alleged) fraud. It was the consistency of results from poll to poll to poll that alerted the researchers to the possibility of fraud.
The thing that surprises me most about both this and the strategic vision scandal is that they probably could have pulled it off if they just invested in a random number generator. Given half an hour I could come up with fake data for a poll that is way more believable than this. It also makes me wonder that if the people who got caught are such bunglers, whether there might be other more sophisticated fraudsters still out there.
Kudos to Markos Moulitsas for taking quick action against R2000. I don’t like his politics, but he’s helping to demonstrate why the blogosphere is better than the mainstream media - it self-corrects. Bloggers fact-check each other. Along the way, the truth filters to the top. For that to work, bloggers have to act the way Markos did when they’re caught in error, even if through no fault of their own.
And more Kudos to Nate Silver at 538, who has earned the respect of people on both sides of the political fence because he’s repeatedly shown that he goes where the math takes him, regardless of his own political ideology.
Check out the link to Benford’s Law that Squink posted. Often numbers that occur in nature aren’t random numbers–for instance the first digit of a number might be much more likely to be 1 than 9, depending on how the number. So a naive RNG is better than just making up numbers, but it still might not provide a data set comparable to real world data.
Aye, and he’s also earned massive respect among fellow statisticians. Last week I was talking to a consultant who specializes in data analysis of fundraising campaigns, who’s published the leading book in the field. Half of our conversation was about Nate Silver and his work in political and sports metrics. The man is taken very seriously, and this incident shows why.
I was just over at Research 2000’s web site. They have not updated their ‘news’ section since April. I do not know just what that means, but it cannot be good.
If we live in an age where at least some pollsters are found to be cheats, who can guess how many were cheating throughout the years? Sort of puts a different light on everything.
