There’s a group that does sidewalk taste tests of bottled vs. tap water in several cities.
The test is described
here. 32 people tested only 3 got 4 cups of water correctly identified.
Does this constitute a valid test? One that deserves special teaser spots and happy chat on all local news shows?
Not enough information. If it was a double-blinded test, the results are more likely to be valid than if it wasn’t, for instance. But, I have to say the name “Assmann” gave me a chuckle.
To expand on QED’s remarks: the study is double-blind if the person running the experiment (offering people the cups of water, tallying their choices, etc) does not know which contain bottled water and which contain tap. This is important: if the person running the study knows what the “right” answers are, he may without intending to cue the subjects to provide the answers he wants (growing subconsciously more excited as the subject considers the answer he likes, for instance).
I can’t find the actual numbers posted anywhere so we can’t check and see if the study is statistically significant, but if tap water and non-tap water are indestinguishable by taste, you would expect (1/2)^4 * 32 = 2 people to guess all four correctly in a study of 32 participants. 3 people did so, according to the article, so that’s about right.
Having the raw data would let us discover some things that the number above can’t tell us. Frex, it could be most people could distinguish Aquafina from tap water consistently but were very bad at distinguishing between Dasani and tap, which would lead to very few people getting all four cups correct, but many more than expected getting 3 out of 4. Similarly, it could be that bottled water is easily distinguishable from non-bottled water, but that there’s no significant difference between Aquafina and Dasani.
Even if everything lines up, though, we can only draw conclusions about the contrast between bottled water and tap water from the tap that the guys who did the study used. The water in the house I grew up in comes from a well, and tastes strongly of iron; it’d be no challenge whatsoever to pick water from that tap out of cups of bottled water. So, their general conclusion (“Bottled water is a rip-off”) only holds if their choice of tap is representative, and no one seems to be talking about that.
So yeah, we need more information about their methodology and data before we can draw good conclusions. Assuming that the guys doing the study aren’t being deceitful (reporting only the 4-out-of-4 statistic because it’s the only one that supports their argument), I think it’s probably scientific enough for a blip on the evening news, but we need some more specifics before we can start demanding ritual suicide from the bottled water makers.
Aside: I once conducted a similar experiment designed to determine whether a Brita filter improves the taste of Pittsburgh tap water (we were all taking a stats class together at the time). One of my two roommates poured 32 cups of water, and the other served me four per day (spaced out throughout the day, to avoid immediate taste contrasts), and reported my results to the first roommate. Each cup was chosen at random to belong to one of four conditions: room-temperature tap, room-temperature Brita, chilled tap, chilled Brita. We chose conditions randomly for each cup to make sure that I wouldn’t subconsciously try to “even up” the results and pick water types I hadn’t seen many of late in the study.
It turned out that I could easily distinguish between room-temperature tap and Brita water, with an accuracy of nearly 100% (I think I missed 2 in 15; it’s been a whlie). For chilled water my accuracy was lower, but still high enough that the difference hypothesis met (IIRC) a 95% confidence level. There was no interesting correlation between correctness and the time of day at which the water was served.
We can conclude from that that Brita water is distinguishable from tap water, at least if your water comes out of our particular tap. To be a really useful study we’d have to repeat it with different taps, but we got bored. To conclude that Brita filters are useful, we’d have to repeat the study with many people; it’s possible that I’m a super awesome tap water detection genius, and that the difference between filtered and unfiltered water, while real, is too minor for those without my leet skillz to detect.
Yes, the test needs to be double-blind or the results are suspect.
I think it’s pretty clear from the article they had an existing bias against bottled water, so I would assume it was not double-blind. (It seems suspicious no representatives of the bottled water industry were on hand.)
Also, unlike bottled, tap water varies in taste throughout the year. I know sometimes tap water tastes fine to me, other times there is a strong chlorine taste. Chlorine levels are adjusted as needed. So you could easily bias the test just by choosing what time of year you sampled the tap water.