The flaws in Randi's tests

If I’m reading you right, Randi isn’t asking the claimant to demonstrate they’re an “omnipotent god”. He’s asking them to demonstrate what they claim to be doing every day. Test conditions are established to prevent any possible cheating.

–Patch

I think your bias is showing, Lib.

Randi and the contestants agree beforehand as to the extent of the power being tested and the appropriateness of the upcoming test to demonstrate that power.

So what’s the problem?

No, I meant what I said: it isn’t as simple as pass/fail to determine authenticity.

If Randi is testing for godhood, he should so state. But if he is testing for psychic ability, then it is like any other ability — subject to failure. For example, I am able to stand on my head. But that does not mean that, in a given test or set of tests, I won’t fail. Maybe the test is unfair, e.g., you want me to stand on my head on a spinning platter. Or maybe I have a migraine that day, and can barely stand up on my feet. Or maybe any number of other factors might hinder the test or skew the data you draw from them.

Can you give an example where Randi has deliberately designed the test to be unfair?

Yes, your abilities are subject to failure. Which is one of the reasons you can take the test again after a period of a year. You can also delay the test if you feel conditons are not ‘right’ (and apparently some would-be testees have done this ad naseum.)

However, what does it say when dowsers are able to find water in pipes when there is no blinding to the test conditions, and then fail when there is blinding?

All the individual tests say id that under controlled conditions, the applicant did not perform as he or she said they would, nothing more. On the whole, the number of failures allows for a stronger conclusion about the applicants.

Then why did you agree beforehand to stand on your head on a spinning platter? As stated before, repeatedly, the claimant and Randi agree to how the test will be performed. If there’s a spinning platter involved in your standing on your head test, it’s because you agreed that you’d do it.

In the Australian dowsing tests that seem to be the focus of these threads, claimants signed a form attesting the following:

“I feel able to perform on this occasion. There are no geographical, meteorological or personal, emotional or physical influences that might inhibit my abilities to perform.”

Claimants also demonstrated that they could perform under the test conditioons. Before the tests began, each claimant demonstrated that they could, indeed, successfully demonstrate their ability when they knew the pipe location.

These baseline tests are standard in tests I’ve read about. Before the tests begin, everything is done to make sure that the claimant is confident in their ability to perform, and are allowed to demonstrate their abilities beforehand. Why? To eliminate the excuse-making that invariably follows failure. “Oh, I couldn’t stand on my head during the test because Saturn moved out of the fifth house which affected the energies flowing from my pyramid-sharpened razor blade and was intensified by the presence of Randi because his skepticism gave me negative vibes.”

If you’re claiming that Randi is requiring claimants to participate in the testing when they state that they’re not able to perform that day, please provide a cite.

–Patch

This reflects an assumption that has been also been put forth by several subsequent posters, but I haven’t seen any basis for it, and it does not seem to make any sense at all. What Randi has claimed is that the test conditions have been agreed upon by both sides. He has NOT claimed that they have been chosen by both sides. And it makes no sense to assume that they have been. Randi would have to be an absolute fool to allow subjects to chose their own rate of success. (Hey I would head over there in an instant and claim a success rate exactly equal to random chance, or lower). Clearly, Randi has a minimum level of success that he will accept as valid.

Of course all the subjects have agreed in advance to criteria of the tests. They obviously have no choice about it - Randi is the guy offering the money. I myself have agreed to the terms of all the tests that I’ve ever taken, but that doesn’t mean that I felt all the terms were fair - it’s just that the only way to take the tests is to agree to the terms.

This is NOT to say that the subjects have a claim on Randi or have been defrauded by Randi because they have no choice in agreeing to the test. Please do not respond by bringing up this point (really, the amount of confusion on this simple matter is astonishing). If I was Randi, I would also insist on control of the terms of the test before handing out a million bucks. But it is misleading to suggest, as RickJay has here, that the claimants had the ability to select a level of success of their choosing - we simply don’t know what minimum level of success Randi would have accepted. And it is a mistake to interpret the claimants’ agreement with the test with approval of the test simply because of the fact that they agreed to take it under those terms.

Talk to me about the actuarial exams sometime… :wink:

No one’s forcing them to accept the test.

In addition, I haven’t seen anyone demonstrate that the success levels required in Randi’s tests are inappropriate. I know for a fact that many people who make claims similar to those Randi spends his time debunking claim very high rates of accuracy and extreme reliability.

The tests ARE extremely unfair! After all, the contestants themselves never agreed to the terms of the test! Oh, wait…

I’m sure that in the end, a meta-analysis of these studies will prove there is something behind it after all. Oh, wait…

In other news, Ludovic has been accepted on the PBA tour on the basis of having once bowled a 268.

But no evidence has ever been presented that Randi has ever asked for an unreasonable level of success. These things aren’t kept private; he’s pretty straightforward about what happens when tests are drawn up. In the dowsing example we know for a fact that the pass/fail criteria seems to have been the dowser’s doing.

If someone has evidence of a case where an applicant suggested a level of success that went beyond any doubt of proof and Randi wanted something ridiculously high that even the applicant admitted could not be done, I have never seen it. If it’s to be asserted that RANDI isd a fraud, then evidence must presented that he is a fruad. So far all we have is assumption and speculation. Show me a test where he set a ridiculous pass/fail against the wishes of the applicant and I’ll start to buy that he might be a fraud.

Furthermore, I’ve seen precious little evidence to suggest that Randi is claiming his tests mean anything more than proving the applicant cannot do what they said they could do. To use the spinning-platter-headstand example, it would certainly be dishonest for Randi to test you as failing to stand on your head on a spinning platter and then say “well, this clown can’t stand on his head at all.” But that simply isn’t what he’s doing; he’s testing people who say they CAN stand on their head on a spinning platter, watching them fall down, and saying “Well, you can’t do that.” I welcome evidence to the contrary. I’ve read the account of the dowsing example several times and I don’t see Randi saying “this experiment proves dowsing doesn’t exist.”

Actually, by all accounts the majority of applicants are resistant to even defining what it is they can do.

Improper analogy. Testing “multiple dowsers” to refute the positive results of a particular successful dowser isn’t the same as retesting the same sucessful dowser.

Whose fault was it that the testing was compromised? To insinuate that the dowser may have cheated without proof or retesting borders on slanderous IMHO.

As for Randi’s testing methods, double-blind testing is a fundamental prerequisite for scientific experimentation. if James Randi is in fact administering these tests (I haven’t seen the NOVA program previously mentioned) that would seem to be a poor testing method which would have more potential to put harm his reputation than his subjects. I don’t see why he’d want to put himself in that position.

James Randi’s ascerbic nature can be a bit offputting even to those like myself and Peter who fundamentally agree with what he’s doing but question his tactics (much in the same way that, though I’m left-leaning politically, I find Bill Maher to be a total jackass.). Heck, even noted scientist and pseudoscience debunker Carl Sagan (or is he also not “scientist-y” enough for the Clarke bashers) found Randi to be a bit overly strident when it came to combatting paranormal claims. (I can track down the cite if anyone’s interested…either in “Demon Haunted Darkness” or “Billions and Billions”.)

Best quote of the thread.

In the Australian dowsing test, to claim the prize, the dowsers had to demonstrate a success rate equal to or greater than their predicted rate. From the preliminary statement, this must be done a significant number of times.

So yes, in that case, the claimants were able to set the bar, but they had to demonstrate that they could perform above that bar at a stastistically significant level.

From the statement they signed, they agreed that:

“I agree that the rules as outlined in the accompanying Document Number Two (Rules for Test) are fair and proper.”

So why sign if they’re not?

From all the published tests I’ve seen, the claimants all agree that the tests are fair, and there’s nothing to inhibit their performance. But once the results are revealed and they’re shown to have failed, suddenly the tests weren’t fair, and excuses abound.

I have never applied for the challenge, so I can’t comment directly on how things go between Randi and a claimant when test protocols are being established. From what I’ve read in published accounts, however, Randi does work to get a mutually agreeable test established that both sides feel is fair. I don’t see the claim that claimants have to agree to tests that aren’t fair as justified.

How long was yours? I took the Civil PE exam - three tests over two days. :stuck_out_tongue:

–Patch

The studio was at fault.

How to you take the statement,

“As far as I am concerned, Mr Cook’s performance on our programme was the result of either his skill as a map dowser, or a remarkable 1-in-twenty-four coincidence.”

as being slanderous?

–Patch

I have no objections to anyone disliking Randi on the basis of his style or acerbic nature. It s amtter of taste.

I do object to false accusations. Peter has made these by insinuating that Randi cheats on tests for financial reasons. Peter has also made other false accusations (such as his attempts to claim that Randi said something he didn’t, in spite of multiple corrections).

Don’t like Randi? Fine, just don’t make stuff up to justify it.

OK, Patch, guess I was a little out of line with the slander accusation. I was just reading between the lines of the …“while he has no proof that he cheated” (but this is how he COULD have cheated)…part. For someone trying to get by on a dowser’s wages, a reputation for integrity is no small matter ;).

Well I can offer you a 99% chance of playing legal moves in a blindfold chess game. Is that level what you call Godhood? Most people can’t even achieve 1%.
From my dowser correspondence:

“we all have it in us to learn to dowse … like most responsible dowsers I charge a fee for my services … we have a guiding rule - never attempt to dowse if you are tired … or unwell … or subject to any worry or anxiety … I know my capabilities, satisfied clients know them, those who I have taught know them”

He sounds pretty confident in his own ability.

As others have said, the tests are agreed to by both Randi and the claimant. So they are palpably not unfair.
Also the claimant is invited to practice beforehand (in the case of dowsers, they are shown where the water is flowing). Only after they are satisfied their powers are working does the test start. So that blows away your ‘migraine’ comment.
What other ‘excuses’ can you come up with?

Here’s the typical test for the money:

  1. The claimant agrees the test is fair.
  2. The claimant practices the test, knowing the answer. They succeed.
  3. The claimant is immediately given a double blind test. They fail.

What conclusion should we draw?

Incidentally you didn’t mention your required level of success, and how many trials you would employ for a psychic claiming to influence coin tosses.
Remember there’s a million at stake - what numbers do you require?

What are a dowser’s wages?
Surely a big oil company will pay a huge sum for instant surveying?
If dowsers are so honest, why don’t they prove their abilities with testing?
If dowswers are so poor financially, why don’t they collect the $1,000,000?

I admit that it is possible to read into things in the manner you did, but I felt that Randi’s comment made it clear that was not what he was doing. Randi outlined several breaches that would permit cheating, but stated that he did not think that was how Mr Cook got his correct result. To give credit to Randi, there was more to his explanation, and to his final comment, than I posted, but I didn’t want to type it all in. I felt that the final comment was enough.

Since the test was done on TV, you don’t have the luxury of time to correct any breaches of protocol (as openly displaying the map certainly was). If this had been some other test format, you’ve more than likely got the time to make a different map to ensure no possibility of peeking.

If anything, I hope that Randi has learned that it is best to have a supply of fake maps about for use in any required rehearsals.

–Patch

I agree that it does seem that way in this particular case, though I imagine Randi must have given a minimum number (or at least would have balked had the number been too low).

Actually Randi has it a bit easy here because most dowsers probably do think their powers are a lot stronger than they are (assuming, of course, that they have any powers at all). And if these guys are in the business of dowsing they would hurt their reputation if they declared their success rate to be too low.

But what I object to is the general implication that the subjects of Randi tests are co-designers of the tests, and that this can be used to prove the fairness of the tests. Clearly this cannot be the case, as noted above.

This is not the same as proving fraud by Randi, as you note. What I am doing, rather, is countering a disproof of fraud that might be inferred from the notion that the subjects and Randi mutually set the parameters of his tests.

As I’ve posted to the previous thread, Randi is clearly implying that his tests (or at least his dowsing one) support his general position that paranormal phenomena do not exist.

Exam? As in ONE exam?

He laughed, a hollow, mirthless laugh…

Anyway, one of my objections to the exam system is outlined here. I think it is unfair. But I took it and agreed to the terms just the same.

Just to emphasise previous comments about Randi’s fairness, here is a report on a dowser going for the million:

http://www.randi.org/jr/032902.html
And some selected bits (bolding mine):

'Mike was to be tested for finding gold, his specialty. The target material he had chosen, and brought along with him, consisted of five quartz stones, nine “Sacagawea” dollar coins, a gold ring, a gold nugget, and a small vial containing water and a few panned flakes. He said he had always been successful in detecting each and all of these seventeen items, when his forked stick was specifically “tuned” to pick up gold — by having a small scrap of gold fastened to its tip. That’s a common claim made by dowsers, that their stick/rod/pendulum is sensitive to the substance attached to it. We decided to use the entire package of seventeen items, sealed in a plastic bag, to give Mike the maximum chance of finding his target, and he affixed his gold “guider” sample to the stick to “tune” it. ’

‘I had asked him to carefully “scan” the floor area of our library in advance to make sure there were no distracting elements present, and he himself carefully chose the positions of each of the ten cups on the floor.’

‘Mike also asked that several metallic objects (trophy cups, plaques, steel devices) be removed from the bookshelves nearby. At his request, a teaspoon was taken to the next room because he said that the silver could also attract his stick;’

‘For the “open” phase of the preliminary test procedure, the target package was placed in the designated cup, which was then openly placed in the spot Mike had chosen for it, mouth-down. He then scanned all ten cups, and declared — both by pointing and verbally — where he believed that his stick had detected the target. Another number was then selected, and the procedure was repeated, twenty times in all. His score was 100% in these “open” tests.’

‘The results were that when Mike G. knew the location of the concealed target (the “open” tests), he obtained 100% results. When the test procedure was double-blinded, he obtained exactly what chance alone would call for: one out of ten correct.’

So that eliminates any unfairness in the tests (the dowser chose the materials and their placements), any problems on the day (the dowser scored 100% in the open tests - equivalent to Godhood?!) and showed an amazing difference when the dowser (and Randi) didn’t know where the material was.

And the scientific conclusion is?

Dowsing is a fraud?

:slight_smile: