Rather than ressurect a semi-zombie from the Pit, I would pose this in GQ.
I like Consumer Reports for some things, and am not a basher. I just notice that they don’t always know what they are talking about as has been hashed over in this thread and I have my theory.
Actually the QM association is just a metaphor, though I hope an apt one. My hypothesis is that CR cannot but help but bias their tests the minute they formulate the conditions of the test.
Let’s say they are testing dishwashers. They are going to establish a test regimen that by its very nature will select the outcome. They will pick a certain type of dish and plate and cup to test in the washer. They will put certain types of food, say chocolate sauce, ketchup, bacon grease, whatever. But what if I don’t have round dishes like they use, or don’t eat bacon grease? I bet I could propose different but equally plausible crap to put on that plate that might change the rankings entirely!
Then there is the whole MPG bias that was seen in the automotive world. It wasn’t long before manufacturers were tailoring gear ratios to the test, sometimes to the detriment of real world economy. Do we really want Maytag to be desiging their dishwashers to score well on chocolate sauce or whatever they use?
So is this the real problem? By the mere act of observation and defining it’s methodology, they are unknowingly changing the results?
I think MPG is infinitely more important to the auto industry than CR rankings are to appliance manufacturers. I would not be at all surprised that automakers tune the cars to pass the EPA mileage tests, but I’d be quite surprised if a dishwasher manufacturer did the same to get a better CR rating.
I think it’s quite possible that one methodology would favor a particular model more than another in terms of say what foods to put on the dirty plates, but I don’t really see the alternative. You have to put something on the plates, and as long as the person selecting what to dirty the plates with doesn’t have knowledge of how that would affect the results, then what more can you ask?
The key is that they TELL you what tests they use. If those tests don’t match your criteria, you’re at liberty to ignore the results. Example:
I’m allergic to most soap. When they test soap, they throw the hypoallergenic types in with the rest, and they always come out low in the ratings. I don’t buy Dove because it’s highly rated, I buy Aveeno because my needs are different from what they’re testing.
The basis of all scientific testing is the attempt to objectively list the test conditions and objectively list the outcomes. This almost invariably involves simplifying the problem because the number of real world factors and their unknown interactions would make trying to replicate them impossibly long, complicated, and expensive. That’s one reason why no one experiment is ever accepted as “the” answer.
The OP is misusing the uncertainty principle, except in the loosest, most metaphoric sense. The CR testing is designed to best duplicate actual use, as far as it can be simplified to a set of test cases. You might be able to come up with certain tests that would give differing results, but probably not a whole suite of them that would be realistic. Is the testing imperfect? Sure it is. Is anyone out there doing anything better? Doubtful. If anything, the testing is too thorough. They take so long to run tests that all too often the particular products tested are no longer on the market.
The CR testing is a guide, and should be used as one factor in a decision. If their tests don’t match your particular usage then it’s not a very big factor. But they go into stores anonymously and buy the products like just the rest of us so there’s little chance that manufacturers can bias the mechanics just for them. They never know what tests will be run in the future.
While it’s debatable whether Consumer Reports’ rankings influence appliance manufacturers, more and more colleges & universities are basing decisions on how it will affect their U.S. News & World Report rankings. So it does happen.
FWIW, teachers face the same problem too - you have to make the exams representative of actual knowledge, and make major changes every year so students won’t just study for last year’s exam.
BTW, in QM, the issue is that the act of measurement changes the actual value you’re trying to measure. I don’t think it’s a very good analogy. (It is a good analogy for my blood pressure though: every time a nurse straps a sphygmomanometer on my arm, I get really nervous and my blood pressure goes up.)
True. But there’s nothing scientific about the U.S. News & World rankings, so manipulating them is a different kind of event. It’s more like a feedback response.
You could say the same thing about tv networks trying to increase their ratings or really any manufacturer listening to consumers’ desires.
Ive used Consumer Reports for many years. I find it a very good starting place when making a big purchase. They describe the process they use,so you arent in the dark. I think the low ratings for American cars have helped the Auto Companies which sufferered from arrogance for many years. They refuse advertising and wont let their results be used in ads. What more can you expect and where else can you go.