Although I have had a pretty good level of training in statistical analysis and research methodology, I am perhaps missing something here. Perhaps some other Dopers can catch any flaws in my train of logic:
A test or series of tests which checked any claim against a control of average statistical likelihood would in fact show whether or not a claim deviated form what one would expect from random guessing.
However, even showing that someone’s "psychic power " was as effective as guess-based/chance-based controls, wouldn’t necessarily prove that their claims were bunk. It could be possible that some “psychic power”, while real, only had an efficacy equal to the control. (As an analogy, if we have a new drug and find that it works as well as a placebo, that doesn’t necessarily mean that the drug never works for various individuals depending on body chemistry and such, only that it doesn’t work better in general than a placebo).
So even a Fail result on a test would not disprove someone’s claims of “psychic power”, it would simply show that they were unable to prove them and/or that their ability was no more effective than random chance, and thus, pretty much useless.
This would of course leave an escape hatch for wooians to scurry out of.
The epistemological basis of using and constructing a null hypothesis, and the methodological basis by which a NH is either held or rejected is thus at issue.
I believe that a proper NH for a ‘psychic power’ would be “any claimed technique or techniques, ability of abilities, and/or method or methods that the claimant has stated they use to obtain result X , has/have an efficacy no different than the statistical likelihood of arriving at the same results by pure chance.”
I do not, however, feel that a proper NH would be “any claimed technique or techniques, ability of abilities, and/or method or methods that the claimant has stated they use to obtain result X, do not exist outside of the claimant’s imagination, and they are either deluded or a fraud.” The problem with this, of course, also being that we can’t know for sure what method someone uses, whether it’s cold reading, hot reading, remarkable subconscious powers of observation and intuition, “psychic power”, or invisible Information Beams from the cloaked Xist motherships in orbit around the planet. Without being able to know for sure how someone endeavored to do something, all we’d know is that whatever they claimed to do either worked better than blind statistical chance, or didn’t.
For instance, we know that quantum entanglement is a real and verifiable phenomena which may even violate the laws of relativity as it appears to operate FTL . We don’t know how, exactly, this happens. We know something is happening, but the exact causes, processes and dynamic aren’t known. Or as Sir Eddington said about Uncertainty “Something unknown is doing we don’t know what.”
Even knowing that someone can do something at a better rate than blind luck would allow doesn’t tell us how they did it. If someone who was a past master at picking up subtle cues was able to score above average at guessing facts about people, but we couldn’t know what process they actually used to get their Hits, we wouldn’t know if they used intuition or the Xist Information Beams. To further complicate the issue, individual statistical results that are sufficiently above the average are also, themselves, part of random chance. Toss a coin 100 times with a person guessing, and the average number of correct guesses will average out to 50 with enough trials. But, with enough trials, the possibility that someone, somewhere, will guess 100/100, rises. And if we perform the trials enough time, the chance that someone can guess 100/100 tosses, and can replicate that feat 100 times, rises as well. Nor is it impossible if we took only one person and only tried it with them 100 times. It is hugely unlikely, but not impossible. So even once we’d confirmed that someone was better at predicting something than blind chance would allow over a large enough distribution of results, that doesn’t mean that their individual results still weren’t the result of blind chance.
We can most properly talk about probabilistic, not absolute, truth values. Even things that are highly unlikely to occur via blind chance (guessing 100/100) are just highly unlikely via blind chance, not impossible.
Again this complicates the challenge and allows wooians to scurry out the back door if they get a Fail result, and to make overstated claims if they happen to be the lucky wooian who guesses 100/100 that day.
These things are unavoidable if we follow proper methodology and interpretation of data.
We can, of course, still point out that we cannot prove a negative in many circumstances. The dynamic is the same as The Dragon in the Garage. If someone claims “supernatural” events and cannot prove them, while the burden of proof rests squarely on their shoulders of the claimants, being unable to prove their claims does not mean that their claims have been disproven… only that there is no rational justification for believing that their claims were true. And in cases where someone alleges an unproven thing is doing something that they haven’t proven, they have to test both parts of the claim. That is, “there is something called the supernatural” has to be tested. “I can do something more reliably than random chance would dictate.” needs to be tested. And “that supernatural thing is what allows me to do something more reliably than random chance” needs to be tested. (yeesh)
And being unable to prove the first or the third assertions does not disprove them, it simply shows that they remain unproven or perhaps unprovable.
Properly, something that can neither be proven nor refuted is classified as “unknown”, “unknowable” or “meaningless”, depending on context.
I see this whole complex of facts as something that would give some wooians enough room to scurry and escape, and a lack of understanding of the burden of proof and epistemology would still limit the Challenge’s effectiveness in the eyes of those who haven’t learned about research methodology. The issue isn’t a simple one, at all.
Do I have any errors in the above statements?
P.S. I would prefer, if people actually want to discuss this sub-issue, that we stick to discussing it. That is, if there was someone who showed no real interest in epistemology or methodology, and that person was to add worthless comments to this discussion, we could all ignore those comments for the good of the discussion instead of getting bogged down in trying to debate someone who isn’t actually engaging in debate, but monologue.
Fair?
