Rather than ressurect a semi-zombie from the Pit, I would pose this in GQ.
I like Consumer Reports for some things, and am not a basher. I just notice that they don’t always know what they are talking about as has been hashed over in this thread and I have my theory.
Actually the QM association is just a metaphor, though I hope an apt one. My hypothesis is that CR cannot but help but bias their tests the minute they formulate the conditions of the test.
Let’s say they are testing dishwashers. They are going to establish a test regimen that by its very nature will select the outcome. They will pick a certain type of dish and plate and cup to test in the washer. They will put certain types of food, say chocolate sauce, ketchup, bacon grease, whatever. But what if I don’t have round dishes like they use, or don’t eat bacon grease? I bet I could propose different but equally plausible crap to put on that plate that might change the rankings entirely!
Then there is the whole MPG bias that was seen in the automotive world. It wasn’t long before manufacturers were tailoring gear ratios to the test, sometimes to the detriment of real world economy. Do we really want Maytag to be desiging their dishwashers to score well on chocolate sauce or whatever they use?
So is this the real problem? By the mere act of observation and defining it’s methodology, they are unknowingly changing the results?