The flaws in Randi's tests

I am a sceptic, let me make that clear from the start. I do not really believe in dowsing (except- possibly- by the getting-clues-from-the –local-terrain method, and I’m even doubtful of that.) If someone wishes to claim they can dowse, its up to them to prove it. I am open minded enough to allow a fair test, but really James Randi is not the man to test their claims. Here’s why.

I first saw Randi on his show on British TV about 10 years ago. It was called “James Randi, Psychic Detective” I watched it, hoping to see some scientific tests of psychic abilities. What I actually saw was fundamentally wrong on so many levels. Here are some of the things that struck me:

The bias of the tester: Any test where the tester has made up his mind as to the result beforehand is seriously flawed.

Failure to follow double-blind procedures: in one test Randi showed a table with a number of items on it. The items were carpentry and DIY tools. He told the audience that four of the items had been involved in a crime, then invited a psychic to divine which ones. The psychic picked four items. Then Randi smugly informed her that all her choices were wrong. He had been in her presence throughout the test.

Failure to give the audience full information. In the above test he never bothered to tell us which items were correct, what the crime was, what the involvement of the objects, or anything. This matters. Did the psychic fail to detect the saw that Randi bought from a hardware shop on a Sunday (which was actually a crime in the UK at the time), or did she fail to detect the knife that was a murder weapon? This is a significant detail, that Randi didn’t bother telling us.

He was too quick to jump to conclusions: he set the subjects just one test, and if they failed he claimed this proved they were frauds or deluded fools.

He was inconsistent.  During the series I recall only one person that passed his test. This was a map dowser, who correctly divined the location of an object on a map divided into about 100 squares. Randi congratulated him, and then he claimed that it was probably just a lucky guess, and one test proves nothing. In all the other tests, Randi was happy to say that one test proved something.

He frequently changed the premises of the test, testing people for thing other than what they said they could do. If someone says they can detect an underground river, he will invite them to detect a four-inch pipe, that sort of thing.

In short, I didn’t think that the tests were fair, or had any scientific validity to them. Not that I believed the claims of the psychics he tested at all, in fact most of them seemed crazy.

The biggest problem with Randi’s tests is that he tests intensively, rather than extensively.  That is he gives a very small number of tests, and demands a perfect or near perfect score. A good example may be found here:  http://www.skeptics.com.au/journal/divining.htm Note the conditions of the test. Subjects were given a choice, 5 or 10 tries and no more. This limit was set by Randi, take it or leave it. He would have liked to have done more, but was sure that his subjects wouldn’t have the patience. Setting the 10 tries was entirely for their benefit, whether they liked it or not. How considerate of him. According to http://skepdic.com/dowsing.html  the pass mark was 80%, i.e. subjects had to get 4 hits out of 5 tries to win the prize. If they managed to get that, they had to do the test a second time in order to claim. If someone had got 4 out of 5 first, but only 3 out of 5 the second time, that would have been a failure, by Randi’s standards.

Given the nature of the test, a 1 in 10 shot, you would expect success in the region of 10%, give or take a percent. In fact, in the water test, subjects scored 22%. This fell far short of the requirements for Randi’s prize, but on the other hand is more than double the expected chance result. The probability of this happening by chance is less than 1%. In fact, this would happen only once in 107 tests. Now this is an interesting result, its high enough to be statistically significant, but low enough to be a fluke. I think it probably WAS a fluke. A person genuinely interested in testing dowsing ability would have repeated the test, several times. If the subjects had scored 20% or more on ALL the tests, then that would exclude fluke. But Randi never does that.

 To get a group score of 22%, one individual must have got 30% or more. Just imagine for a moment that this is a genuine, albeit limited psychic ability. Imagine that he could score 30% every time he took the test.  Could he pass Randi’s test? Could he get 4 out of 5, twice running? On his best day he would have to be very, very lucky indeed to pass. 

Randi fails to give us the individual scores for the dowsers: he only gives us the group total. It might be that all subjects got 2 out of 10, and one got three out of 10, but on the other hand it might be that one got zero, while anothern got 5. I want to know what the highest score was. Randi chooses not to share the information.

In his commentary on the test Randi includes two false arguments. First of all, he lists the dowsers’ expectation of their scores, and points out that they had predicted an 86% success rate, and fell short of it. Yes, they were over-optimistic about their own level of skill, but this does not mean a thing. The test was whether they could detect water, not whether they could judge their own abilities. The fact that they did less well than they thought does not change the fact that they did a lot better than chance.

The second is worse. There is a story of the man who put one hand in the freezer, the other hand in the oven, and said that on the whole he felt fine. That man might have been Randi. Randi mentions that, oh by the way, he also ran two other tests, a test for gold dowsing, and a test for brass dowsing. Randi takes the 0% score on the brass test, adds it to the 22% score on the water test, and declares an overall result of 13% for the three tests together. A test well below chance plus a test well above chance on the whole make <<13.5%, a figure well within expectation.>> according to Randi. This is a mathematical absurdity. These were separate tests, for a different thing, taken by entirely different groups of people. Only TWO people were involved in more than one test. They have nothing to do with each other. Combining the scores gives no meaningful result. The result in the water test was 22%, not the 13% Randi implies.

Randi uses the test results to support his anti-dowsing assumptions. This is wrong. The 22% result in the water test may be a fluke, or it might be an indication of some slight dowsing ability. It certainly does not provide evidence against dowsing, in fact it gives slight evidence in its favour.

Randi’s conclusions are certainly flawed. This is not to say that he should play no part in the testing. Certainly conjurers sometimes cheat in these tests. A magician like Randi has the skill to identify the cheats. But the actual design of the test, and the conclusions from the results should be left to people who know what they’re talking about. 

I am not the only one who sees these flaws in Randi’s testing methods and conclusions. See for example:  http://www.tricksterbook.com/ArticlesOnline/Dowsing.htm << James Randi (1979), professional magician and member of the Committee for the Scientific Investigation of Claims of the Paranormal, conducted a test with four dowsers in Italy. Procedures were spelled out in detail prior to the test and agreed upon by the dowsers. The dowsers were asked to locate three buried pipes with running water and to place pegs over the route of the pipes. As stated by Chamberlin (1980), the test had several deficiencies. No meaningful statistical evaluation was possible. Even if the dowsers had been quite close, they were unlikely to fulfil the requirements for a successful test (they were required to place the pegs in a strip eight inches wide). None of them was able to claim Randi’s $10,000 reward. The test contributed little knowledge to the scientific community. >>   Note that this is not the opinion of a dowsing believer, it is a scientific review of dowsing tests. It is equally critical of many studies that find evidence of dowsing. But this is a frequent response when real scientists take a good hard look at Randi’s tests.

In a case where judging is required, this would be so. However, the test conditions reequire that the results be unambiguous so as to avoid testing. It doesn’t matter what Randi thinks if you bend as spoon with your mental powers.

Dangerous in this case, but it does not exclude the validity of the test. Double-blind is needed to avoid giving clues to the testee. However, this does make for poor television (i.e. Randi nowhere to be seen), so they may have insisted. Randi should have sufficient control to avoid giving clues.

Or it was edited out. I’m sorry but this is just weak.

He’s had a lifetime of expose frauds and the self-deluded. I’d hardly call his expression of such an opinion “too quick”.

One hit in how many tries?

We’ve been over this. It is not true. The dowsers say they can find water in pipes. Randi tests them for that.

So far, you have not supported you viritol against Randi.

[QUOTE]

<snip> I’m sorry, but you’ve made the arguements over the Australian test and the bootom line is: THE DOWSERS FAILED

They did a little better than chance. Not a lot. Had they hit 40% or 50% then you’d have a complaint. As it stands this is just statistcial noise. Keep in mind Randi has tested lots of dowsers, and the results are usually near chance.

22% is still a failure.

[QUOTE}
Note that this is not the opinion of a dowsing believer, it is a scientific review of dowsing tests. It is equally critical of many studies that find evidence of dowsing. But this is a frequent response when real scientists take a good hard look at Randi’s tests. **[/QUOTE]

Just who are these scientists?

You were accusing Randi of lying. So far you haven’t proven anything. In fact all you’ve done is basicly repeat your story with more numbers. WE saw you cloying for information on sci.skeptic and this nitpicking isn’t going to convince anyone.

You complaint about the averages have been brought up before. Rather than repeat what has been said I will just be linking

Mike Hutchinson refutes Peter’s arguement 4 years before Peter makes it.

One.

Arthur C. Clarke for a start, according to your link.

They did very much better than chance.

Then you cannot read. I posted the message to two maths groups and sci.skeptic. The majority of respondants are scientists and mathematicians, and they mostly agreed that there are serious flaws in Randi’s conclusions. Go read the thread http://tinyurl.com/av2o

No matter how often I point out untrue things he has said, you keep desperately twisting his meaning, clutching at the most ridiculous straws to convince yourself he’s being honest. There’s no evidence that will convince a fundamentalist.

That is the basic flaw in Randi’s tests. he does things that make good television or popular books, instead of performing tests with any scientific validity.

Having followed the previous threads on this topic, I’m not prepared to take your ubsubstantiated word on the details of a TV show you saw more than a decade ago. Sorry, but I think you have a clear and fairly irrational bias against James Randi, and I think this colors your perceptions of him. For example, you’re still going on about the pipes v. rivers thing despite the fact that half a dozn people have shown that dowsers do, in fact, claim to be able to locate water in pipes. Bottom line: I don’t trust your recollection of the TV show. Unless you have a cite to an unbiased source for any of these accusations, I’m going to simply disregard them.

Regarding your links: As always, your complaints that the tests are unfair to the dowsers is silly, because all the participants agreed that the test was fair before undertaking it. The dowsers themselves, who should know the extents of their abilities better than anyone else, have all agreed in writing that they are being tested for something that is within the range of their claimed abilities. And then they all fail. If the tests were unfair, then they have no reason to sign the agreement. They’d be agreeing to take a test they know they couldn’t pass, and would only end up damaging their own credibility. Randi’s test called for an 80% success rate. If the dowsers thought they couldn’t achieve that high a success rate, why would they agree to the test? It simply makes no sense for these people to agree, in writing, that the tests are fair if they honestly thought it was unfair.

Your last link, I read long enough to see that it had been originally been published in The Journal of the Society for Psychical Research. Aside for the obvious bias, I’m pretty sure “psychical” isn’t a word. These people can’t even get the terminology for their BS hocus-pocus right, so I’m not going to bother reading their arguments supporting it.

Considering your rather one-sided and inaccurate portrayal of events in the past, and the continued assertion that Randi tests people on things they say they can’t (Among other claims), despite being repeatedly proven wrong, I’m going to require a cite for all of those claims before I’ll believe them.

Your continued pointing to that 22% is simply irational. IIRC, none of the people who got a high success rate on the first run could repeat it on the second, or even come close. Seems to indicate a fluke to me. Meaning, it doesn’t prove dowsing, which is exactly what Randi says; nobody could demonstrate that they were, indeed, dowsing water as they claimed they could. But no, you say that Randi claims that scoring “only” 22% proves dowsing is false. I’d like to see a cite for that claim, because I’ll bet good money that it’s another of your fabrications…

Wrong. You seem to have a fatal misunderstanding of science that you employ to cover your hatred of Randi.

  1. Double-blind tests are neccessary when it is important to remove bias of both the subject and the tester. This is only important when elemetns the test is subjective or plans to break cause some new revelation. Randi’s test was neither of these things. Double blind in this case is preferred, but not critical as long as the tester is capable of avoiding giving hints. Randi is able to do this since he is aware of the data leakage and unsuspecting tester could give out.

  2. You seem to ignore the fact that since this test was single-blind and not double-blind then that acutally works in favor of the testee and against Randi! This little detail seems to have escaped you, especially when you complain about how unfair his testing methods are. He give some slack in the test and you are all over him.

  3. We are also woefully lacking details. You complain that Randi was present, thus not making it a double-blind test. However you do not give any details about what the psychic in question insisted upon. They may have demanded the precense of someone familiar with the objects, thus allow a potential bit of cold reading. By inserting himself, Randi assures that no inadvertant clues are given, while still adhereing to the conditions required by the testee.

Blindly (pun unintended) insisting that there is something automatically wrong with the tests simply because they are not double-blind is the mark of scientific illiteracy. Plenty of tests in science can be performed to single and no blind conditions, depending on what is being tested.

:rolleyes:

I seriously doubt this. Given that you are doing simple recall from something ten years ago…

That’s one, and not a very strong one at that. Clarke wrote the intro to at least one of Randi’s books. Clarke and Randi had some small

But still well within the noise level one might expect to encounter. It is far, far removed from the 80% claimed by dowsers. Sorry, I’m not impressed. Were it consistant, you might have something, but other controlled tests of dowsers have not shown this spike, at least not consistantly. Data spikes like this happen all the time.

Umm, read it and “no”. You’re in your own little world. Some disagreed with the test mixing, but that is irrelevant since that is what the dowsers agreeed to!

Which you have utterly failed to do! The only thing you have come up with is that you found some folks who might think it was unfair to mix the results. Don’t care, no win either way.

As you prove here with each post. You’ve got diminishing returns on your claims, and yet you still want us to villify Randi on the basis of your memories and some underperforming dowsers.

I don’t think so.

Rubbish.; I have an entirely RATIONAL bias agaist Randi.
To quote Arthur C. Clarke, an internationally reknowned scientist from the link posted by Miskatonic <<What are we to make of Randi’s test for dowsing? The experiment was well-designed but I don’t quite agree with his conclusions. The test for water and the test for metal were entirely separate experiments. He shouldn’t have combined the results.>> I figured that out for myself without Arthurs opinion.

It is because I kept seeing things like this in Randi’s work that I dislike him.

No I do not. I point out that his tests are flawed. Y^ou claim that the flaw works in favour of the testee, so you are going to ignore it. Flawed is flawed, however you twist it. If it is in favour of dowsing, that does not change the fact that it is flawed.

Anbd you are wrong, anyway. A tester can influence a subject through things like body language either way, either making it more likely they will pass, or more likely they will fail.

Actually, since I don’t believe in dowsing, and map dowsing least of all, I think it likely that the subject was picking up on Randi’s body language.

wrong

in your dreams.

I have given a lot. The fact that you so desperately twist Randi’s words, trying to pretend that he never denied the existence of undergroud rivers says more about you than about me.

well, lets see. Agreeing with me, I’ve got Arthur C. Clarke, internationally reknowned scientist. Agreeing with you, you have …?

I don’t want you to vilify him, or anyone else. I just want you to see that his tests are flawed, his conclusions are suspect, and his motivations are financial.

No, you have misunderstood. Read again.

There were three SEPERATE groups of people.

<<Eight tried for the water via the buried grid, three tried for brass and seven tried for gold… Two of the contestants did two kinds of tests, for water and brass.>>

NOBODY agreed to mixing water and gold.
NOBODY agreed to mixing gold and brass.
TWO out of NINE agreed to mixing water and brass. The other seven did not agree.

NOBODY AT ALL agreed to mixing water, brass and gold, yet that is exactly what Randi has done.

And Arthur C. Clarke, internationally reknowned scientist, has said that’s wrong.

No, you have misunderstood. Read again.

There were three SEPERATE groups of people.

<<Eight tried for the water via the buried grid, three tried for brass and seven tried for gold… Two of the contestants did two kinds of tests, for water and brass.>>

NOBODY agreed to mixing water and gold.
NOBODY agreed to mixing gold and brass.
TWO out of NINE agreed to mixing water and brass. The other seven did not agree.

NOBODY AT ALL agreed to mixing water, brass and gold, yet that is exactly what Randi has done.

And Arthur C. Clarke, internationally reknowned scientist, has said that’s wrong.

You are missing a basic point here. Randi DID NOT give them a second run. He should have done so. Instead, he fudged the figures to make it appear they had done less well than they actually did.

Couild have fooled me.

Yes, adn you still don’t get it that * it doesn’t matter!* Combined, seperate, mixed, pureed. All tests combined or seperate still failed to show signifigance.

I rather doubt it.

Neither test was flawed, Peter. There was nothing wrong with either design.

Yes they can. However, the key word here is control. If Randi is able to influence the testee via body language, then what are they claiming to do? I doubt seriously that their claim was “I can use a person’s body language to pick objects”?

The point is a controlled test. Not slvaish obedience to terminology.

:rolleyes: You had this explained to you in painful detail, and you still insist upon clinging to this delusion. Nobody, and I mean nobody agreed with your interpetation. Random people entering the thread saw that you were in error.

Appeal to authority noted. Clarke is simply wrong. None of the tests, combined or otherwise, came to a win. A spike in the data means nothing.

Emphasis added.

So now you are accusing him of fraud in the same sentence that you claim you don’t want to villfy him. Suuuuuuure.

No YOU are missing the point. Combined or otherwise the tests were a failure. A faiure, get it? They announced the water results first and they were a failure! Randi is under no obligation to keep testing them until a statistical fluke (that is still fraction of the success rate claimed by the dowsers) goes away.

You may note that Mike Hutchinson, who is an authro and scientist (though not as popular or reknown as Clarke) disgrees with most of Clarke’s interpetaions. You chose to ignore their proper criticism he gave, inclduing stating what should be obvious to you by now:

The entire thread that Mike’s comment appears in shows how illegitimate this claim of impropriety is.

The odds were more than 100 to 1 against.

Most people, including Arthur C. Clarke consider that significant.

And YOU dont get it, Randi combined the tests to disguise the significant results he got. That is at best a mathematical error, and more likely a deliberately misleading account on his part.

Oh, an internationally reknowned scientist is wrong, while Randi, who admits his lack of scientific knowledge, knows more than him.

Riiiiiiight.

So , by your definition, anyone who comes to Randi and says they can make a 1 in a 100 shot should be elligible? That’s pretty unfair to Randi.

How could he disguide what was announced? Pray tell? They announced the results to everyone. They still failed.

And these terms were agreed to by the dowsers. Randi didn’t just pull an eleventh ohour stunt to get a test result below 80%. They did not succeed. randi didn’t just throw this bit in, it was part of the agreed conditions.

He claims that 22% is a win for the dowsers, he is wrong. No matter what his credentials. You are doing what is called an “appeal to authority”. Clarke 's main objection seems to be that Randi did not change the test conditons agreed to by the dowsers to fosuc on a higher, but dismally failing grade.

Say I take a multiple choice test with 5 answers , where passsing grade is 70%. I choose to just put int random choices (chance 20%). By a fluke, (1-in a 100) I get say 42%. Should I be given a passing grade? I don’t think so.

Now, if I were to continue taking tests like this (as Randi tests multiple dowsers) and continue to see this fluke, there might be something to it. But there has been no sign of this in being anything more than a fluke of statistics, one which you seem determined to blame Randi for.

I gotta agree… randi is almost definitly right and these people aren’t psychics… but alot of his tests are sort of iffy as to if they are actually testing something similar to the power the person claims to have.

I’m not sure its reasonable to assume that a power to point in the general direction of a thousand gallons of underwater water would translate into being able to point to the exact location of a small amount of water.

of course I don’t really think these people have any powers at all, but I am consistantly unimpressed with how randi tests, he always seems to do tests to test for a much much stronger and more powerful talents than the person ever seemed to claim to have.

That’s a nitpick with Randi’s extrapolations from the results of his tests. It says nothing about the validity of the tests themselves, and ignores the fact that no matter how you slice the data, the dowsers still failed the tests.

Does it bother you at all that no one else on the planet sees these things?

You claimed that Randi was a fraud. That is to say, that he rigs his tests to make it impossible for his subjects to pass. Since it’s now been shown that if anything, his tests are rigger to favor the subjects, and they still fail, can I expect a retraction of the accusation of fraud?

Ah, no, actually body language just makes it more likely that they will pass, since the danger is that an experienced con artist can read his mark for subconscious body language. If the subject isn’t looking for body language (in other words, isn’t a conartist) than body language will have no effect on the test’s outcome. As has already pointed out, this means the test was broken to favor the subjects. And they still failed, which shows that this particular defect was immaterial to the validity of the test.

Could be, or the guy just made a lucky guess.

This is flat out and deliberate dishonesty. First, all the dowsers agreed in writing to the conditions of the test, including the use of PVC piping instead of “underground rivers.” The dowsers all agreed that they would have no problem detecting moving water in pipes, and many (if not most) dowsers worldwide claim to be able to detect water (among other things) in pipes as well as in natural streams. Furthermore, Randi’s comment about underground rivers was unambiguously refering to a specific claim made by many (if not most) dowsers, including the ones involved in that test, that underground rivers are a common phenomenon in all types of terrain. There is absolutely no scientific basis for this theory, and that is precisely what Randi was refering to when he said, “Besides, the “underground river” notion that dowsers maintain is sheer fiction, not supported at all by geological research.” This theory is described, in detail, by a dowser, on this website, and this theory is repeated on several other dowsing websites. And, just as Randi says, there is no support in geological research to maintain this notion maintained by dowsers.

Between myself and others, an easy dozen links were provided in this thread that clearly document both the dowsing in pipes issue, and the underground rivers issue. Interesting that you never returned to that thread to address any of the cites you had requested. Hopefully, it’ll have the same effect in this thread.

Everyone on Earth who isn’t an octogenarian science fiction author in Sri Lanka.

Again with the lying. Or do you think that calling someone a fraud isn’t “vilifying” him?

Okay, two things you really need to learn:

  1. Being a scientist doesn’t make you infalliable.

  2. Being popular doesn’t make you right.

So knock it off with this “Internationally reknowned scientists Arthur C. Clarke” crap. It’s not impressing anybody.

Christ, don’t encourage him!

I’m not sure it’s reasonable, either, but the dowsers themselves had no problem with it: they claimed at the outset that they’d have no problem detecting small amounts of water in plastic pipes. Even signed a paper to that effect.

Can you provide a cite for him doing that? Might as well ask you, since peter morris apparently is never going to come through with one. Every test I’ve seen Randi do has been under conditions the subject agreed to in writing. In other words, he tests exactly what they claim they can do, and to make sure there’s no question about it, he has them write down exactly what they can do.

I’m afraid that until you come up with some way in which his tests lack validity, instead of just throwing enormous amounts of unciteable mud, the peripheral red herring of Randi’s motivation is an irrelevancy.

I am particularly amused by this:

First you say that ACC is an “internationally reknowned scientist”. Then you quote, presumably with approval, a passage in which ACC says, “the experiment was well designed”.

Now, peter, check out the title of this thread.

It seems you are all in favour of “argument from authority” but you only seem to take any notice of “internationally reknowned” authorities when they agree with you, but are blind to their contradiction of your primary point.

Snort.

Is it time to run off to a new thread yet peter?

I’m pretty sure that on every math, chemistry & physics test I’ve ever taken, the professor had his mind made up as to the proper result beforehand. Were these tests “seriously flawed”?

No, if the claimant says they can detect an “underground river”, he will ask the claimant if they can detect water flowing through a pipe just as well (so that the results of the test can be fairly judged without having to go digging for “underground rivers”). When the claimants respond to the effect that they can detect X amount of water flowing through burried pipes, Randi supplies 2X that amount, just to make sure.

With regard to the dowsing tests in Italy, have you seen the surveyor’s map of the burried pipes, and the map of the dowsers’ results? It was so far off as to be laughable.

Why are you upset that Randi gives X number of tests? Why aren’t you just as upset that the claimant agrees to X number of tests?

Yes it does. Randi askes the claimants “what can you do?” Claimant says “I can do X.” Randi says “fine, do it.” Claimant fails to do X. Claimant protests, saying “but I did Y and Z!”. Can you see how tiresome this sort of thing would become after a while?

No, the test was whether or not they could detect water as often as they said they could. Heck, I can detect water. Just don’t ask me with what kind of success rate, that way I can weasle out of the claim if the results come back less than stellar.

Have you asked him for it? Can you show us where he has been asked for the information and is on record as refusing to share it?

I’m with Miller. What’s the deal harping about a ten year old show? Is that the most recent thing you can find that irritates you?

I support James Randi’s efforts to debunk those who claim to have special powers, but from what I’ve seen of his work he is a better showman than scientist.

One test that I recall (sorry, no cite - I’ll see if I can find something later), Randi gave a group of students an astrolgical reading telling them that each reading had been prepared for each individual student by an astrologer. After the students had read their “personal” reading they were asked how closely it came to describing their personality. For the most part they said that they were amazed how accurate the readings were. At that point Randi disclosed that all the readings were identical.

Randi went on to claim that that test debunked astrological claims. The problem that I have with the experiment is that it really didn’t test what Randi was claiming it did. If he claimed that people are gullible when it comes to vague descriptions of who they are, then fine, but it really wasn’t putting astrology per se to the test.

Rats, I really need to get going. I’ll have to come back and flesh this out. Sorry.