Agreed. But the point is that this can only be done by people who understand maths.
The real problem occurs when a test is conducted by someone with no background in maths, who claims that agreeing a pass mark in advance is a valid substitute.
But nobody here has been suggesting that the math is irrelevant. A lot of the discussion has been on the statistics.
As I understand it, the suggestion that the definition of success be agreed to in advance is to lock the dowser in to a definition of “success”-which is impossible if he doesn’t agree to it.
Either (1) the dowser won’t agree to a definition of “success” that is statistically different from chance, in which case his “ability” is no better than guessing, or (2) he says that ** he can achieve a specified level of success **, determined by basic probability and statistics–so that when he fails to do so, he can’t fall back on the “justification” that the test was too harsh, or that he got some hits, and those ought to be viewed as evidence—stuff that is, mathematically, nonsense, but can be very persuasive.
Just as I discuss above in positive controls–one key of this kind of testing is to have the “ability” defined in advance–what can it detect, with what ability, and so on. The test has no force if it determines the dowser can’t do X, and he is able to argue that he never said he could do X, only X’.
In fact that is exactly what a lot of people claim, in previous threads on this forum.
Sure - some quick Googling produced this:
Well, that is a problem–it’s terrible when people don’t understand math–but it’s not a problem with the posts you’re responding to. And that’s all I’m trying to do–address the arguments you are making, in the context of the posts you’re replying to–and in that context, nobody is suggesting just making the numbers up. It’s just not a problem with the protocol we’re discussing here.
If folks in some other thread say math doesn’t matter, then complain about it in that thread.
Slightly off-topic, but if a dowser claims to only be able to find naturally-occurring underground water, then there’s still a way to test it. In short, the idea is to ask the dowser to find a location where there is not water. The reason so many people believe dowsing works is that you can dig a hole almost anywhere, and eventually find water.
If you want to perform a more rigorous test, then you can divide a field or whatever up into, say, 100 smaller regions, and ask the dowser to find the places where the water is closest to the surface. Then you dig wells in all the locations, and see if the places closest to the surface are where the dowser said they were.
Amusing related story.
sweeteviljesus since you mentioned someone was asking for examples of tests that showed dowsing doesn’t work that story has a link to the US Department Of Justice stating exactly that in regards to “dowsing for bombs”.
No, if you look, you will see that I was responding to an individual in **this **thread who stated “everyone has to agree in advance how many trials will be performed and what number of hits constitutes a positive outcome.”
He is actually referring to a specific test. It gets cited a lot on dowsing threads on this forum. The dowsers scored 22% when by chance alone they should have scored 10%.
People who understand maths have looked at that result and declared it to be a positive result for the dowsers. But the man conducting the test and the dowsers had agreed a score of 80% in advance. There are a lot of dopers who claim that the agreement overrules the laws of maths.
I should point out that the guy conducting the test has an average intelligence, no scientific background, and little education of any sort. Almost certainly the 22% score is due to some mistake on his part.
The “agreed in advance” malarky is something he has invented as an excuse when he gets it wrong, which happens fairly often.
Yes, I see that. It’s immediately under his description of the percentage by which the dowser claims to beat chance, and statistically determining how many trials are necessary to confirm a claimed effect of a given scale. I haven’t checked his math, but it seems to be a discussion of fairly simple statistical concepts.
I don’t see that at all. I see him discussing how to design an experiment–and doing so in a way with clear criteria at the start.
Cite? Again, I see no discussion of specific, already performed experiments in this thread by people other than you. The OP is discussing experimental design–not previously performed experiments. If there is some subtext going on, I’m not getting it–and honestly, don’t have the time to search through the entire forum to find a reference to something that is at best, tangential in this thread.
Really? There are, in fact, quite good reasons for doing so. I and others have pointed them out. Further, James Randi, who is immensely experienced with designing protocol for such tests, also demands the level of success to be agreed in advance–for the very reason pointed out in this thread–to make it hard to “draw targets around where the shots hit”, and to make it easy to evaluate the result of such testing.
Again, I don’t see where you go from “define success in advance” to “ignore the laws of mathematics in doing so.” When I have seen such tests, the “target” is calculated using two numbers: (1) the claimed ability of the dowser, and (2) the level of significance required–how unlikely it must be to get such a result by chance. Again, I don’t see any problem with that. The dowser can declare whatever he wants for (1), and for obvious reasons, gets no input into (2).
That is not “making it up.” The purpose of agreeing on it is to ensure that both parties are happy with the test protocol ahead of time, so that the outcome is clear–so that one side can’t dispute the protocol after the test is completed.
That is a claim often made by opponents of dowsing, but it is not correct.
James Randi has only got immense experience with getting it wrong. And the reason for the “agree in advance and ignore what mathmeticians say” is to provide himself with an excuse to cover his blunders.
Not only does he get it wrong, but blunders too? How terrible. Well, obviously, you’re hoping that just saying so means that I will discount everything he says immediately, ignoring the conclusions I have reached from what I have read about the tests he does. That would be one possibility.
While I completely believe you think his methodology is wrong, I take the view that, in a discussion of how to design an empirical test, one instead would expect to see support for such claims, and if it wasn’t present, it would be quite reasonable to request specific references to examples of the phenomena you’re describing, and a description of how those examples support your assertions.
Or to put it another way, cite please? If you’re right, it would be immensely helpful to the OP to see how such a test ought to be done. Care to enlighten us? I am, in fact, fairly well educated in both probability and statistical analysis, so no need to simplify on my account.
Fine, I’ll give you a specific example. But first, let’s clarify a couple of points. Best if we get this agreed in advance, right?
In an earlier post you said:
So, would you agree that a **statistical **analysis of the results is important?
Would you agree that any test *without *a statistical analysis is worthless?
Would you agree that it should be someone trained in statistics that should set the “success” level?
Would you agree that it is silly to ask a dowser without mathematical training to set the success level?
Hypothetical case: In a particular test, a statistician says that a score of 30% or over is significant. A dowser claims to have an 80% success rate. In fact, he takes the test and scores 35%. Which of the following is true?
- he passed, because he got more than the statistically significant mark
- he failed because he got less than he said he would.
One more question: in a dowsing test, what would be a good way of randomizing the location of the target? Say it was one of ten positions.
That would be me.
Huh? I have no idea what you are referring to. I used the OP’s idea of a test using 10 containers and that is all. Period. I didn’t even google “Dowsing tests” or anything similar.
I have no idea what test you are referring to and I’ll give you the benefit of the doubt by assuming that “he” and “malarky” and the other comments are directed at somebody out there who has actually run some kind of test, however if those comments were directed at me you owe me an apology.
Depends on the particular physical setup but let’s suppose the testee agrees that he can detect a gallon jug of water sitting under a cardboard box. Put ten cardboard boxes on the floor and number them 1-10. Roll a ten-sided die (30 cents at your local hobby shop). Put the jug under that box. Person who does all this then goes someplace where they cannot see the rest of the experiment.
Testee comes in and does his thing, indicates which box he thinks contains the jug. To be ironclad about it, his prediction is written down. Boxes are then all lifted to show which one concealed the jug. Record whether trial was a “hit” or not.
Repeat for designated number of trials.
Do all this on film to avoid anyone saying after the fact “He cheated and peeked under the boxes” or “I didn’t say it was under #7, I said #3” or any such stuff (on either side).
Regarding significance levels - I specifically did not set some magic number and say “Here’s your standard”. That is obviously a question for a qualified statistician, but I still say that both sides need to agree beforehand to accept the results, whether they be positive or negative.
Well, one has already been presented.
The only necessary modification to this is that EACH box should be moved after each trial–so there is no way of detecting the full box by looking for the one that’s in a slightly different location.
Another simple method (for running water)–requires some preparation, but pretty foolproof.
Bury three pipes in a 30’ by 30’ area (perhaps a sand pit, where it would be easy to bury pipes and then rake the area over. Ideally, each pipe should have at least 2-3 curves–not be a straight line.
Set up a place that can be shielded from view, with (1) a inlet pipe (or pipes, if you don’t have (2)), and/or (2) a set of valves to allow the flow into one and only one of the three pipes. Label each inlet pipe with a letter , and make a map of each pipe’s location.
Have someone completely different come by, and replace the labels on the inlet pipes with numbers, assigned randomly and recorded.
So now person #1 knows the routes of each pipe, and which pipe corresponds to which letter, and person # 2 knows which letter corresponds to each number (with which the valves are now labelled).
Tell #1 and #2 to go far away.
Bury a length of straight pipe in the presence of the dowser, exactly as was done to the “test” pipes. Without water in it, invite him to examine it and to verify that it reads as “empty.” Repeat with a flow of water through it identical to that used on the test, to ensure he can detect a known positive.
Have person #3 take a fair die to the inlets. Run three trials. In each trial, the water will be poured into the pipe corresponding to the die roll (out of sight ofthe dowser). Nobody but #3 will know which pipe was filled. The dowser will be invited to mark the route he detects water in in each trial (perhaps with small flags). Record their locations on a map.
Then, compare the recorded locations to the full pipe in each test (getting #1, #2, and #3 together to identify which pipe was filled in each trial).
They were not directed at you.
They were directed at a certain individual who you cited.
Your points don’t have anything to do with your assertion that randi’s methodologies were both wrong and full of blunders. If you have specific problems in mind, with a specific study, point them out. If not, retract the point.
I think it is not silly at all to ask a person claiming to have a certain ability to specify what he claims to be able to do. If he can’t do that, how can you possibly test it?
Has it? You mean throwing a dice?
I was under the impression that dice were mechanically biased, and can’t produce a truly random number.
Don’t you think that this might slant the test?
I think you misunderstand – they have to agree in advance WITH what the mathamaticians say. If the math says that 5 is a success, the dowser has to agree in advance that 5 is a success, and can’t say later, if he only got 4 right, that it was “close enough”.