Why would that be the final say? Homeopathic remedies, healing prayer, therapudic touch, astrology, chiropractic subluxions, magnetic therapies, and so on and so on cannot possibly work, have no science underlying their supposed processes, and fail consistently when properly double-blind tested.
And yet you’ll get thousands of people willing to tell you how they were cured by one of the above. Many of whom will have actually BEEN cured after the treatment in question.
People get better all the time, without intervention at all, even of often-fatal diseases. They will tend to attribute their cure to the last thing they tried. People who don’t, no matter how many there are of them, fail to report their findings.
I had a grandfather that lived to be very old, despite weighing over 450 pounds. We all know folks that lived into their 90’s while chain smoking. These are the EXCEPTIONS, not the rule, and we’re biologically wired to pay more attention to to exceptions than “normal” processing. So we need a way to filter out this inherent human bias.
Please. If you never listen to another thing I say, please listen to this:
THERE EXISTS A PROCESS BY WHICH WE CAN DETERMINE TO WHATEVER DEGREE OF CONFIDENCE WE LIKE WHETHER A TREATMENT IS CORRELATED WITH A DESIRED OUTCOME.
This process can’t be stymied by the FDA, the government, the scientific community, men from mars, or industrial conspiracies. It doesn’t have to cost a lot of money.
It does require some effort, a large enough population of subjects with the condition you want to treat, and a commitment to being non-biased.
That process is the randomized, controlled, double-blind (RCDB) trial. All those words are important:
Randomized: The subjects need to be assigned to treatment and control groups in a manner which doesn’t bias the outcome, usually computerized random number generation these days, but you can flip a coin if you like.
Controlled: There needs to be an identical (or as close as possible) group of subjects who don’t get the treatment (possibly given a placebo), in order to verify that any effects seen in the treatment group aren’t just normal processes of the condition.
Double-blind: Neither the people giving and receiving the treatments, nor the ones deciding what outcomes are “success” may know whether an individual was in the control or treatment group (at least until after the experiment and results are complete). This is the one that the famous prayer study on heart conditions failed to do, for example.
You can’t leave out any of these without re-introducing the human tendency to see what you want to see. Randomized, controlled, double-blind studies produce data. Your coworker telling you that JiffyVite cured her cold is an anecdote. You can show anything with anecdotes, whether or not it’s true.
When you do a proper study, you get two things: A result, and a confidence level in that result. At this point you can say things like: yes, I’m 95% sure that JiffyVite decreases the length of the common cold by 30% on average. Incidentally, it has no effect at all on 60% of those who take it, and a much greater effect on those for whom it works.
Why can you say that? Because the result has been actually demonstrated, in an environment that takes out human bias. What’s absolutely, amazingly stunning to folks who first start looking at the scientific trial method is how many effects that you KNOW are true…aren’t when you start actually looking at them. Our capacity to fool ourselves with non-representative data is astounding. This “confirmation bias” is probably the strongest psychological effect present in human reasoning, and we’re largely unaware of it.
Don’t like the confidence level? Fine, do a larger study. You can pick any number less than 100%), and statistically choose a sample size that will give it to you. It’s always a good idea to do more studies anyway, because that can expose errors in control, randomness, or blinding.
Bottom line: Be sceptical of any claim that hasn’t had an RCDB trial done on it. If it has, ask whether the randomization method is valid, the control group was really selected randomly and is similar to the test group, and especially whether blinding was maintained all the way through the experiment. If so, and you still don’t believe the answer, test it yourself. The whole point of this process is that it can be reproduced, (and done properly should give the same answer), WHETHER OR NOT THE TESTER BELIEVES THEY ALREADY KNOW THE ANSWER.
RUN AWAY from anyone that claims that RCBD data can’t be done on their product for some reason (“the skeptic mental energy necessary for RCBD ruins the results” is popular), or worse, that it was done, but the negative result was wrong for some reason (the proper response there is to do the experiment again, not to explain it away).
Yeah, I know you can’t do this for every claim you come across. But people hawking “miracle cures” never seem to do it (or leave out important steps, usually either the control or blinding ones). If you can pass such a study, and it confirms your effect, the scientific community WILL take note (although if they’re skeptical, the first thing they’ll do is replicate it).
Thanks to anyone who read this far, sorry for the longwindedness. I’ll now go back to trying to solve people’s computer problems.
