Sweet! That means Obama is really up by at least 4.
No, that does not follow.
Consider just this:
One polling firm consistently reports THE most accurate results because they have sample sizes twice as big as any other firm.
Assuming everything else is equal, are results of pooling that firms results with a dozen other firms’ data, creating an n an order of magnitude larger, going to be less accurate, less reliable?
The problem is that everything else is not equal. There are house effects. Some firms are not as accurate because of smaller sample sizes but because of flawed methodology. Taking into account those factors in a statistically sophisticated and partisanly impartial manner is what gives Silver his edge over straight up aggregators.
Problem is, I don’t think it’s fair to assume that. If we’re postulating a world where every pollster is exactly the same, save for the sample size, then my question would be “why in the heck do we have so many polling firms out there doing exactly the same job?”
It’s not just sample size that sets one firm apart from another. So it’s not possible to simply mush them all together and come up with a more accurate number due to a greater sampling than any one polling place.
So I’d ask the question again, if you’re Rasmussen (or any firm that thinks they’re the best) why would you believe that mixing your data together with an inferior pollster’s data would produce superior results to that which you alone could obtain?
I suspect that the real value of Nate’s model is that it gives a much better estimate of the uncertainty in the races than any of the competitors. Unfortunately, that’s not something that you can really see without running simulations, and you can’t really sell that to the general public.
I think every smart pollster understands that there’s a margin for error in his or her poll, and that on any given day, some other poll may show a more accurate depiction of the state of the race then theirs (even if they still thinks theirs is best).
Also, I think that historically, poll averages are better predictors overall then any particular individual poll (I don’t have a link for this).
That follows pretty much immediately from basic statistical theory. The question being discussed is whether a straight average is best, or if you can do better with a clever weighting scheme. The answer is that the latter is almost always better.
As of today the only difference between 538’s electoral results and RCP’s no toss ups map is VA, which Silver rates as 67% going Obama and predicts a 1.2% margin, and which RCP states as leaning to Romney by 0.5%. Hence RCP no toss up is at 290 Obama and 538 is at 303.
For Senate Silver states “52.4” to “47.6” compared to RCPs no toss ups of 53 to 47. They seem to differ on Indiana (RCP placing it as a D win and 538 calling it R). RCP has 11 races as toss ups.
Ender, did you read my complete post? Weighting those inequalities in a statistically sophisticated manner without partisan bias is what Silver brings to the table.
The response though is to the logic you offered: “THE most accurate” individual will necessarily be more accurate than pooling together with a host of others all less accurate than they.
Let’s even imagine another scenario - A thousand decent archers shooting at a target that no one can hit exactly dead center. Each gets one shot. Which is more likely to be closer to bullseye, THE best archer of the batch, or the average of the shots of the other 999 good archers, even if a few tend to pull right, some left, some up, some down? How about if you can identify which archers always pull up or down and by about how much and factor that in as well?
I think, in theory, that actually the vast majority of weights will leave you worse off than equal weighting. However, in most cases there’s a more accurate weighting scheme out there that needs to be determined/discovered. Worst case is that you can simply use equal weights as your default weighting scheme. Finding this weighting scheme is the challenge.
Edit: Actually I guess that’s what you meant with ‘clever weighting scheme’. Never mind, then.
No, I agree with you. Which is why I think Nate Silver provides a service. Realize that my whole argument stemmed from OMG’s belief that Nate Silver was in the Democrat’s pocket, that his results were just okay, and that they were equal to or worse than Rasmussen’s.
So I’ve been trying to argue the reverse. That if Rasmussen’s really that good, there’d be no need for people like Nate Silver. Heck, if other aggregators were that good, there’d be no need for Nate Silver.
I disagree with this. The statement is only true if the variation in the different polls is due to standard fluctuation, in which case you’re simply increasing the sample size by including more polls. But to the extent that different polls use different methodologies, some of which may be wrong, you’re not necessarily increasing the accuracy by including them, and certainly it does not follow from basic statistical theory.
Only if done properly.
No one really knows precisely what Silver is doing, in his various adjustments. I would guess that he’s probably doing a good job overall, but this is not a known fact.
Is there any other way to evaluate whether Silver is doing a good job then to look at how his predictions have compared to election results? Because on that metric, he’s been very good.
Yes, you can analyze at the model based on known statistical concepts.
Ultrafilter is a statistics guy, and his comments seemed to be made in that context, and I responded accordingly.
I agree that NS has had pretty good results so far, but it’s over a fairly short period. Most likely he’s doing a very fine job, but I don’t know that it’s conclusive to this point.
The core problem with arguing that you should follow only one firm is that you can’t know how good a poll is until after the election.
No matter what the track record of the firm has been in previous years, you can’t be sure that it’s superior this year as well.
In the real world, no firm has sufficient time or money to allow it to be better than everyone else in every election in every year. And the real world deficiencies of polling ensure that even the firm with the most time and money can’t be perfectly accurate. If you’re the customer then you have to pick the polling firm that you think best suits your needs. But those polls are not the ones which are made public. For everyone else conglomerating all the polls and all their different methods gives a better picture than any individual poll.
You can’t be sure of any one poll. You can’t be sure of any one polling outfit. Looking at averages and following the trend line is far superior.
You do realize that consistency is not necessarily a good thing, right? It’s possible that the '08 race was actually quite volatile, and Rasmussen tracked very closely to that actual volatility, while other pollsters that showed a steadier, more consistent race were consistently wrong for the duration of the race. Unfortunately, the only chance we get to measure how accurate they are is on election day, so we just don’t know.
:rolleyes:
This seems like crazy talk. How do you figure that? Why 4? Why not 2, 16, or 60?
Awesome! Obama’s up by 60!
Theoretically, yes. This thread is the first I’ve heard, though, that Rasmussen starts off with more bias and gets closer to reality come election time. I want to see more evidence of that before I buy it. There may be nothing to explain at all.
Rasmussen doesn’t use cell phones - they robocall - and their results are typically 2-3% more Republican than the average of the other polls. Cite.
We have consistent bias (from the other guys, not necessarily from God’s actual value for Χ), we have a (non-evil) mechanism that reasonably accounts for that bias. I’m willing to conclude Rasmussen polls are biased.
But if they’re the most accurate, doesn’t that mean that everyone else is biased?
Yes.
I think that Silver claims that the bias is historical though, that is that in showed the same bias in previous elections. If that ends up not being the case this time, the model will need to be adjusted.
The most significant thing about the latest Rasmussen is that it moved a couple percent towards Obama from their previous poll.