# The mandatory "Million Dollar Challenge is a fraud" thread **URL:** \ **Category:** Great Debates\ **Created:** [February 11, 2005, 6:37am UTC](https://boards.straightdope.com/t/the-mandatory-million-dollar-challenge-is-a-fraud-thread/289555 "2005-02-11T06:37:56Z")\ **Posts on this page:** 20\ **Page:** 12
**Author:** ![Aeschines](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/aeschines/32/16251_2.png) [@Aeschines](https://boards.straightdope.com/u/Aeschines)\ **Post date:** [February 16, 2005, 7:46pm UTC](https://boards.straightdope.com/t/the-mandatory-million-dollar-challenge-is-a-fraud-thread/289555/221 "2005-02-16T19:46:08Z")
> [@Abe](#): > > It seems the item being latched on to this particular iteration of the same old argument is “what is statistically significant”? I don’t have an argument about that: for that answer I simply turn to the science of statistics. > [@](#): > > Perhaps we can take a look at specific examples one by one instead of hand-waving about how such strict standards aren’t ever followed in the social and medical sciences. Distortion. I never said strict statistical standards weren’t applied in medicine and social science (they are not as strict as in the hard sciences, but that doesn’t mean that I think those standards are not appropriate for those sciences). What I said was that, by Randi’s arbitrarily high standards, the phenomena discovered in those sciences would be rejected. > [@](#): > > **Aeschines** , you, the departed **Snakespirit** , and **Peter** “JREF killed my dog” **Morris** have been equivocating about standards of evidence on the topic of the paranormal for years now; but what will it accomplish? Raw sophistry. I ain’t them, they ain’t I, and my arguments are mine and, IMO, correct. Speak to the arguments. > [@](#): > > Even if you establish that JREF screwed up on a couple times (Kramer sure sounds like a weird guy to have on board) that won’t automatically render the applicants endowed with paranormal abilities and win them the prize. Irrelevant. Nowhere in this thread have I said that Randi “screwed up.” I am talking about the theory behind his standards. > [@](#): > > Here in this lengthy wrangle, you questioned the fact that anecdotal evidence is unacceptable in science, but that in the social and medical sciences anecdotal evidence is actually acceptable. Different argument. Nowhere in this thread have brought up the issue of “anecdotal evidence.” Speak to the arguments here. > [@](#): > > It would seem so. The FDA, for example, routinely fails millions of dollars’ worth of experimental drugs every year on the basis of unacceptable statistical significance in its trials. We wouldn’t have developed a clue on how to treat serious diseases over the last century if we were truly unable to establish which treatments proved successful in a statistically meaningful way. This is an irrelvent point, as I was never questioning the FDA’s standards to begin with. Get thy logic in gear, fellow. ---
**Author:** ![SentientMeat](https://avatars.discourse-cdn.com/v4/letter/s/43a26b/32.png) [@SentientMeat](https://boards.straightdope.com/u/SentientMeat)\ **Post date:** [February 16, 2005, 8:23pm UTC](https://boards.straightdope.com/t/the-mandatory-million-dollar-challenge-is-a-fraud-thread/289555/222 "2005-02-16T20:23:01Z")
> [@](#): > > A p-score of .05 would mean that there is a 1/20 chance that the results occurred simply due to chance. Those aren’t astronomical odds against chance. Which is why drug tests run _multiple_ tests, _each_ having such strong confidences, and the results **combined** to give .05 x .05 x 0.5 _etc._, since otherwise **1 in 20 drugs will have been approved merely on lucky evidence**. Do you really think this is true? Maybe more than 1 in 10 billion drugs _do_ get approved so, but such an incidence is _nowhere near_ as common as that. ---
**Author:** ![Princhester](https://avatars.discourse-cdn.com/v4/letter/p/3e96dc/32.png) [@Princhester](https://boards.straightdope.com/u/Princhester)\ **Post date:** [February 16, 2005, 11:10pm UTC](https://boards.straightdope.com/t/the-mandatory-million-dollar-challenge-is-a-fraud-thread/289555/223 "2005-02-16T23:10:45Z")
> [@Aeschines](#): > > Distortion. I never said strict statistical standards weren’t applied in medicine and social science (they are not as strict as in the hard sciences, but that doesn’t mean that I think those standards are not appropriate for those sciences). What I said was that, by Randi’s arbitrarily high standards, the phenomena discovered in those sciences would be rejected. Why so? Do you think those phenomena fluked it, and that if you just ran heaps more trials, they’d eventually be proven to be statistical blips? I’m still far from sure you get it. ---
**Author:** ![Peter\_Morris](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/peter_morris/32/359_2.png) [@Peter\_Morris](https://boards.straightdope.com/u/Peter_Morris)\ **Post date:** [February 16, 2005, 11:41pm UTC](https://boards.straightdope.com/t/the-mandatory-million-dollar-challenge-is-a-fraud-thread/289555/224 "2005-02-16T23:41:39Z")
I think that YOU are the one that doesn’t get it. If Randi was running medical tests, he would try each drug on 5 subjects, and demand that 4 of them were completely cured after 1 dose of medicine. The result must be obvious to any observer. No messing about with expert medical opinion making a subjective judgement that a patient’s condition has improved. Only complete cure will do. Any drug failing that test would be rejected. That is the same way he runs his telepathy tests. ---
**Author:** ![Bryan\_Ekers](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/bryan_ekers/32/183_2.png) [@Bryan\_Ekers](https://boards.straightdope.com/u/Bryan_Ekers)\ **Post date:** [February 17, 2005, 12:20am UTC](https://boards.straightdope.com/t/the-mandatory-million-dollar-challenge-is-a-fraud-thread/289555/225 "2005-02-17T00:20:54Z")
> [@Peter Morris](#): > > I think that YOU are the one that doesn’t get it. > > If Randi was running medical tests, he would try each drug on 5 subjects, and demand that 4 of them were completely cured after 1 dose of medicine. The result must be obvious to any observer. No messing about with expert medical opinion making a subjective judgement that a patient’s condition has improved. Only complete cure will do. Any drug failing that test would be rejected. > > That is the same way he runs his telepathy tests. That’s rather a stretch, isn’t it? The only way the analogy makes sense is if the drug companies have made the claims of insta-cure (or something else that seems inherently implausible, much like paranormal claims)and Randi evaluates those claims. If the drug company wants to make the looser claim of simple “improvement”, the standards of the test must also be loosened. Trouble is, “loose” paranormal claims (“I can do this minor feat, some of the time”) are hard to evaluate and, in my opinion, not worth a million bucks. ---
**Author:** ![Princhester](https://avatars.discourse-cdn.com/v4/letter/p/3e96dc/32.png) [@Princhester](https://boards.straightdope.com/u/Princhester)\ **Post date:** [February 17, 2005, 12:48am UTC](https://boards.straightdope.com/t/the-mandatory-million-dollar-challenge-is-a-fraud-thread/289555/226 "2005-02-17T00:48:06Z")
> [@Peter Morris](#): > > If Randi was running medical tests, he would try each drug on 5 subjects, and demand that 4 of them were completely cured after 1 dose of medicine. And if that is what the drug company said their drug could do, why would that be inappropriate? ---
**Author:** ![Aeschines](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/aeschines/32/16251_2.png) [@Aeschines](https://boards.straightdope.com/u/Aeschines)\ **Post date:** [February 17, 2005, 12:56am UTC](https://boards.straightdope.com/t/the-mandatory-million-dollar-challenge-is-a-fraud-thread/289555/227 "2005-02-17T00:56:16Z")
> [@Princhester](#): > > Why so? Do you think those phenomena fluked it, and that if you just ran heaps more trials, they’d eventually be proven to be statistical blips? I’m still far from sure you get it. Non sequitur. It seems that you’ve taken my point, however: that the odds Randi requires cannot be applied to medicine and the social sciences. That’s the point, not whether the standards in the social sciences and medicine are actually inappropriate. If I were to have an opinion, it would have to be given study by study. ---
**Author:** ![Peter\_Morris](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/peter_morris/32/359_2.png) [@Peter\_Morris](https://boards.straightdope.com/u/Peter_Morris)\ **Post date:** [February 17, 2005, 1:04am UTC](https://boards.straightdope.com/t/the-mandatory-million-dollar-challenge-is-a-fraud-thread/289555/228 "2005-02-17T01:04:27Z")
> [@](#): > > That’s rather a stretch, isn’t it? Not really. As we have seen, Randi has a tendency to write people’s claims for them. B.C. made a claim that works only 1/3 of the time, and Randi refused to test it. She is forced either to drop out and abandon her claim - in which case Randi declares victory by default. Or else she has to change her claim and say she can do it 100% of the time. I’ve read claims by various paranormalists, and it’s rare that they claim 100% accuracy. ---
**Author:** ![Princhester](https://avatars.discourse-cdn.com/v4/letter/p/3e96dc/32.png) [@Princhester](https://boards.straightdope.com/u/Princhester)\ **Post date:** [February 17, 2005, 1:14am UTC](https://boards.straightdope.com/t/the-mandatory-million-dollar-challenge-is-a-fraud-thread/289555/229 "2005-02-17T01:14:07Z")
> [@Aeschines](#): > > Non sequitur. It seems that you’ve taken my point, however: that the odds Randi requires cannot be applied to medicine and the social sciences. No, you miss my point. If the scientific establishment demanded that medicine and the social sciences meet higher odds, testing would just be done to those standards. Nothing would get inappropriately rejected. > [@Peter Morris](#): > > B.C. made a claim that works only 1/3 of the time, and Randi refused to test it. This is the only objective part of the relevant paragraph, the rest is citeless editorialising. The only objective part of your paragraph doesn’t amount to much. > [@](#): > > I’ve read claims by various paranormalists, and it’s rare that they claim 100% accuracy. Indeed. How is this relevant? ---
**Author:** ![Peter\_Morris](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/peter_morris/32/359_2.png) [@Peter\_Morris](https://boards.straightdope.com/u/Peter_Morris)\ **Post date:** [February 17, 2005, 1:15am UTC](https://boards.straightdope.com/t/the-mandatory-million-dollar-challenge-is-a-fraud-thread/289555/230 "2005-02-17T01:15:09Z")
> [@Princhester](#): > > And if that is what the drug company said their drug could do, why would that be inappropriate? Sigh. **Drug company :** We’ve got a drug that works on one patuient in three. **Randi :** That’s less than chance. It’s not medicine. \*\*Drug company : \*\* Wha? One person in three is cured. None would be cured without our drug. **Randi :** Yiou have to show a higher rate of success to count as medical. \*\*Drug company : \*\* But we can show you statistics that our drug works far more often than chance. **Randi :** One out of three is less than chance. You must show four out of five to count as proof. repeat until drug company gives up and goes away. ---
**Author:** ![MysteryFellow63427](https://avatars.discourse-cdn.com/v4/letter/m/a88e4f/32.png) [@MysteryFellow63427](https://boards.straightdope.com/u/MysteryFellow63427)\ **Post date:** [February 17, 2005, 1:21am UTC](https://boards.straightdope.com/t/the-mandatory-million-dollar-challenge-is-a-fraud-thread/289555/231 "2005-02-17T01:21:40Z")
> [@Aeschines](#): > > It seems that you’ve taken my point, however: that the odds Randi requires cannot be applied to medicine and the social sciences. Randi does not necessarily require “high odds” for his Million Dollar Challenge. However, the “looser” the paranormal claim, the more trials are required to verify that the claim is being met. If I claim to be able to predict the results of a (fair) ten-sided die roll twenty percent of the time (twice as often as chance), would you believe me if you rolled the die five times and I was correct once? Of course not, because I might have just gotten lucky. Do that same test a thousand times, and if I’ve predicted 200 rolls correctly after that, I’ve got a pretty good claim. Do that same test a hundred thousand times, and if I’ve predicted 20,000 rolls, you can bet your bottom dollar that it isn’t just luck. The more trials you do, the more confident you can be of the results. The reason medical trials and social science studies can’t always achieve the higher confidence intervals that Randi may demand is in part due to cost. Studies and patient trials are expensive. However, the cost of doing repeated telepathy trials (for example) should be relatively cheap. Part of the reason that there needs to be a high confidence interval is chance. If Million Dollar challenge tests were set up with only a 96% confidence interval, it wouldn’t take long before someone got **lucky** and won. The real moral of the story is that stastically speaking, the weaker the claim is the more tests that need to be done to prove it. If you made an extremely weak claim (“I can predict the results of a coin flip 51.3% of the time!”), you’d need oodles of trials to eliminate sheer chance and support your claim. (I jumped into this thread in the middle, so please forgive me if some of what I’ve said is redundant, I don’t have time to go back and read the entire thread right now.) ---
**Author:** ![Aeschines](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/aeschines/32/16251_2.png) [@Aeschines](https://boards.straightdope.com/u/Aeschines)\ **Post date:** [February 17, 2005, 1:26am UTC](https://boards.straightdope.com/t/the-mandatory-million-dollar-challenge-is-a-fraud-thread/289555/232 "2005-02-17T01:26:24Z")
I worked in the drug industry in marketing for two years. I’m not an expert on how they design their tests, but I can comment insofar as I have knowledge, which is not an insignificant level. > [@SentientMeat](#): > > Which is why drug tests run _multiple_ tests, _each_ having such strong confidences, and the results **combined** to give .05 x .05 x 0.5 _etc._, since otherwise **1 in 20 drugs will have been approved merely on lucky evidence**. Do you really think this is true? Somebody needs to start a “Bad Statistics” site. A. You run “multiple tests,” yes, on multiple persons. You get, in effect, one large sample. The concept here of “multiple tests” makes no sense in the context of drug research (sure, if a company can’t prove efficacy with the sample size they have, which often happens, they may increase the sample size with a “new test”; but if the protocols are the same, then it’s still the _same test_. B. What’s really egregiously wrong is your idea that the p scores of “multiple tests” could be multiplied by each other to get a multiplicative effect. In reality, the opposite is true: To narrow the confidence level you need progressively bigger samples, and it is a geometric progression, not arithmetic (reduction in confidence x requires current sample size_y_). That’s why in political polls they stick with smallish samples and a margin of error of 3%; to get that error down to 1% or so would require mammoth samples. If my knowlege were fresher and better, I could explain the above better. Nitpick away. But the basic principles are correct, whereas what you wrote was completely off-base. > [@](#): > > Maybe more than 1 in 10 billion drugs _do_ get approved so, but such an incidence is _nowhere near_ as common as that. By now it’s clear that you don’t know what you’re talking about. But let me add that many drugs do go on the market whose efficacy is later doubted, and a few make it onto the market that are completely unsafe. These things happen precisely because the sample sizes are _not_ big enough, and both drug companies and governments _knowingly take the risk._ Once the drugs are used by a larger sample (i.e., millions of actual users in the public), then the truth comes out. ---
**Author:** ![Czarcasm](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/czarcasm/32/4050_2.png) [@Czarcasm](https://boards.straightdope.com/u/Czarcasm)\ **Post date:** [February 17, 2005, 1:27am UTC](https://boards.straightdope.com/t/the-mandatory-million-dollar-challenge-is-a-fraud-thread/289555/233 "2005-02-17T01:27:02Z")
Lets see now-six pages, no evidence of fraud on Mr. Randi’s part whatsoever, and not one legitimate paranormal claim. And I was afraid that this thread might deviate from all the other ones on the same subject. 😃 ---
**Author:** ![PatriotX](https://avatars.discourse-cdn.com/v4/letter/p/c4cdca/32.png) [@PatriotX](https://boards.straightdope.com/u/PatriotX)\ **Post date:** [February 17, 2005, 1:33am UTC](https://boards.straightdope.com/t/the-mandatory-million-dollar-challenge-is-a-fraud-thread/289555/234 "2005-02-17T01:33:28Z")
> [@Czarcasm](#): > > Lets see now-six pages, no evidence of fraud on Mr. Randi’s part whatsoever, and not one legitimate paranormal claim. > > And I was afraid that this thread might deviate from all the other ones on the same subject. 😃 Has there been a suggestion from JREF objecters as to what would constitute an acceptable testing practice? ---
**Author:** ![Peter\_Morris](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/peter_morris/32/359_2.png) [@Peter\_Morris](https://boards.straightdope.com/u/Peter_Morris)\ **Post date:** [February 17, 2005, 1:34am UTC](https://boards.straightdope.com/t/the-mandatory-million-dollar-challenge-is-a-fraud-thread/289555/235 "2005-02-17T01:34:58Z")
> [@Princhester](#): > > Indeed. How is this relevant? The fact that Randi doesn’t test the claims people make. He twists their claims, and then tests for \*\*his \*\*own twisted version. ---
**Author:** ![KidCharlemagne](https://avatars.discourse-cdn.com/v4/letter/k/ed8c4c/32.png) [@KidCharlemagne](https://boards.straightdope.com/u/KidCharlemagne)\ **Post date:** [February 17, 2005, 1:46am UTC](https://boards.straightdope.com/t/the-mandatory-million-dollar-challenge-is-a-fraud-thread/289555/236 "2005-02-17T01:46:22Z")
> [@Czarcasm](#): > > Lets see now-six pages, no evidence of fraud on Mr. Randi’s part whatsoever, and not one legitimate paranormal claim. > > And I was afraid that this thread might deviate from all the other ones on the same subject. 😃 I don’t think this thread is concerned with whether the paranormal exists or not. I think that there may only be one person arguing against Randi who really believes in the paranormal. Similarly, I don’t think many of us remaining in the thread are really claiming fraud, just not acting in good faith. No pun intended. ---
**Author:** ![Princhester](https://avatars.discourse-cdn.com/v4/letter/p/3e96dc/32.png) [@Princhester](https://boards.straightdope.com/u/Princhester)\ **Post date:** [February 17, 2005, 1:54am UTC](https://boards.straightdope.com/t/the-mandatory-million-dollar-challenge-is-a-fraud-thread/289555/237 "2005-02-17T01:54:10Z")
> [@Peter Morris](#): > > Sigh. Poor old Peter. You’re just so put upon by us dolts. How do you ever cope? It’s just so hard being the only person smart enough to see things as they really are, when surrounded by fools. It makes me sad, just feeling sorry for you given what you have to put up with from us. Sigh. Sniffle. Sigh. But my sincere sympathy aside, the problem with your dramatisation is that you made it up. The resemblance to the BC case is only passing. And I still don’t get the relevance of the fact that paranormalists rarely claim 100%. What percentage do they claim, usually? ---
**Author:** ![Peter\_Morris](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/peter_morris/32/359_2.png) [@Peter\_Morris](https://boards.straightdope.com/u/Peter_Morris)\ **Post date:** [February 17, 2005, 2:04am UTC](https://boards.straightdope.com/t/the-mandatory-million-dollar-challenge-is-a-fraud-thread/289555/238 "2005-02-17T02:04:32Z")
> [@PatriotX](#): > > Has there been a suggestion from JREF objecters as to what would constitute an acceptable testing practice? Well, Randi’s usual test is : _You must score 8 out of ten, or 4 out of 5 to pass_. Instead of that, have enough trials to make a valid statistical analysis possible. Say about 50 trials in a test. See if the subject can score \*significantly \*better than chance. Say, a 98% confidence level that it’s not fluke. If he passes the first test, test him again to see if he can replicate it, or if it was a fluke. Repeat the test several times, so that the results either revert to chance levels, or you can be 99.999% sure a paranormal effect has been displayed. ---
**Author:** ![Princhester](https://avatars.discourse-cdn.com/v4/letter/p/3e96dc/32.png) [@Princhester](https://boards.straightdope.com/u/Princhester)\ **Post date:** [February 17, 2005, 2:09am UTC](https://boards.straightdope.com/t/the-mandatory-million-dollar-challenge-is-a-fraud-thread/289555/239 "2005-02-17T02:09:35Z")
In short, Peter, what you are saying is that you think that Randi should conduct research to see if applicants have any paranormal ability, rather than conduct tests to see if applicants can do what they say they can do. Correct? ---
**Author:** ![Mr.Miskatonic](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/mr.miskatonic/32/310_2.png) [@Mr.Miskatonic](https://boards.straightdope.com/u/Mr.Miskatonic)\ **Post date:** [February 17, 2005, 2:13am UTC](https://boards.straightdope.com/t/the-mandatory-million-dollar-challenge-is-a-fraud-thread/289555/240 "2005-02-17T02:13:50Z")
> [@Peter Morris](#): > > Well, Randi’s usual test is : _You must score 8 out of ten, or 4 out of 5 to pass_. > > Instead of that, have enough trials to make a valid statistical analysis possible. Say about 50 trials in a test. See if the subject can score \*significantly \*better than chance. Say, a 98% confidence level that it’s not fluke. And if there is failure of the first 5 trials? Or in the first 10? Must you conintue with the remainder? Statistical analysis really isn’t needed in most cases. Particularly when what you are dealing with is a demonstration. [Previous page](https://boards.straightdope.com/t/the-mandatory-million-dollar-challenge-is-a-fraud-thread/289555.md?page=11) [Next page](https://boards.straightdope.com/t/the-mandatory-million-dollar-challenge-is-a-fraud-thread/289555.md?page=13)