Halloween season seems an appropriate time to mull over this recent spooky research. (Here’s a link to the original article, in press. Haven’t read it all yet. Kinda lengthy.)
Assuming the methods and results are sound, can anyone think of an explanation that does not seem to violate one or more fundamental laws of nature?
Nothing is ever proven till it’s been replicated by others who think it’s a load of crap.
Simplest and, I think, most likely explanation: The researcher is lying.
Next-simplest explanation: The researcher thinks he’s telling the truth, but considered a lot of the trials no good for some (apparently) valid reason or another, but was more likely to find a reason to discard the trials that showed no effect.
Next simplest explanation after that: It was a pure fluke, and the random error just happened to be stronger than usual in the direction of showing an effect. Remember, when a study says something’s significant at the 95% confidence level, what that means is that even if there’s no effect at all, 1 time in 20 you’d get a result about like the one you got.
Next explanation after that, there’s some glitch in the computer programming for the “random” selections that biased the computer towards giving the after-the-fact help to the people who did better.
I could go on, but I think you get the point: There are a very large number of explanations which need to be ruled out before “psychic precognition” becomes the best remaining explanation.
Probably the best one-line description of the scientific method that I’ve ever seen.
Really?
Yeah, I try to keep that in mind as well. I recall V. S. Ramachandran mentioning a conference with 30,000 papers and abstracts, and I’ve wondered whether that means that about 1,500 bogus results were presented at that conference.
Nonetheless, it’s food for thought.
Most Published Research Findings are False, even ones that do not contradict almost everything else that we know.
Absolutely. A really good scientist runs the experiment again trying to convince himself it is a load of crap and that he did something wrong - but no one is perfect, and sometimes we fool ourselves.
As a former conference program chair, I assure you that any conference with 30,000 papers has more than 5% that are bogus and crap. I’m getting nightmares even thinking about this.
What percent of papers present results at the .95 confidence level? What percent require .99 or higher?
Without that information, the statement that “when a study says something’s significant at the 95% confidence level, what that means is that even if there’s no effect at all, 1 time in 20 you’d get a result about like the one you got” is meaningless for evaluating research despite its abstract truth (pun intended).
.95 is the commonly accepted standard for publication.
95% is the commonly accepted minimum confidence to accept a result as meaningful. That means all results reported as significant will have a confidence greater than 95%. Many results will have a much higher confidence still – it’s not uncommon to see 99%, 99.9%, or 99.99%.
That said, there’s absolutely a lot of bogus results out there, due to stacked biases at every level.
You want spooky? I’ll give you spooky. About forty years ago, my father and a grad student were goofing around with a physiograph one night in the psych lab at UCLA. The physiograph was set up with both EEG and GSR transducers. My father shined a series of lights into the grad student’s eyes, and he repeatedly measured a galvanic skin response of 100 nanovolts before the evoked potential. In other words, his skin knew about it before his brain did. Proof of the existence of meridians? I dunno, and my father had no opinion on the matter. To him it was just a weird anomalous thing that happened. But I’d sure like to see if I could reproduce it.
ETA: and no, neither me nor my father is/was trying to yank anybody’s chain. This really happened.
Agreed. As Richard Feynman said (paraphrasing), the easiest person to fool is yourself. As a scientist, I consciously try to design experiments to prove myself wrong, because try as I might, I’m human and often have personal preference for one result over another. I try to think like someone who thinks I am full of shit - “what would I do to show that my hypothesis is crap.”
I just had to have a conversation with one of my post-docs today about the proper use of control. He had a great looking gel. I said “great! Where’s the positive control?”
He said he ran a negative control to control for contamination, but didn’t run a positive. “Who cares, he said. it worked - we don;t need a positive control”
I had to disabuse him of this. “How do we know that these results are real without a positive control? You need to throw this out and re-do the entire thing.”
He won’t do that again. I guarantee it, since i’ll fire him if he does.
Oh, and I took a look at that study. The effect, if one exists, is pretty weak. I don’t believe he’s fabricating data, but I don’t buy it either - I think its just a coincidence. If it can be repeated consistently by unaffiliated neutral labs then i’ll give it more credence, but I’m not holding my breath.
Any paper worth its salt will say what the confidence level is. If they list their confidence level as 99% instead of 95%, then (assuming you trust them), obviously my statement about 1 in 20 is incorrect. But it’s still a 1 in 100 chance of getting those results from chance, and there are probably thousands of folks who have done or are doing experiments on things like this. Even if I trusted absolutely that a researcher’s methods were completely rigorous and absolutely scientifically correct (which of course I don’t), and even if they had a confidence level of 99.99%, it’d still be the simpler assumption that their results were a statistical fluke than to assume that they were correct.
Now, if this effect really is real, the experiments seem simple enough, and there should be no difficulty at all in getting multiple groups to repeat them a great many times. Repeat the experiment enough, and you can get the confidence level as high as you want. So if it’s real, they shouldn’t have any real difficulty in cranking up their confidence level up to, say, 99.99999%. Once they do that, then it’ll be worth paying more attention to. And by “more attention”, I mean at least a half-dozen groups doing the experiment completely independently, to rule out the (very real) possibility of dishonesty, bias, or systematic errors.
here is a link to a relevant article in “The Wall Street Journal”:
I see nothing very spooky about this at all. First of all, the evoked potential is by no means the first response in the brain to a stimulus, it is merely the first one massive enough to be registered on an EEG. There could easily be few a neurons sending a signal from, say, the LGN (where the optic nerve terminates) down to the brainstem to set off a GSR long before the visual cortex had built up enough of a mass response (in thousands or maybe even millions of neurons) to show up as an evoked potential. EEG is a very crude, low-res measure of brain response.
Secondly, was your father very careful that his subject got no cues that the light flash was about to come (such as, maybe, seeing your father move his hand toward the button, or, come to that, just a learnable regularity in the temporal sequence of flashes) before the flash actually arrived? If he did not actually set out to look for this sort of effect (as you seem to say was the case) he may well not have been making great efforts to conceal such cues, and very subtle cues of this sort often get picked up (often unconsciously) by experimental subjects if they are not very carefully and deliberately concealed. The GSR could have been a response to the cue - tensing up in expectation of a flash, with the evoked potential being a response to the flash itself.
^ Yip, you actually tend to predict everything before you actually get the information. When the prediction is off, you often lose reaction time.
A common example is that of a batter thinking they can keep their eye on the ball the entire time.
OK, this is from a very quick read through, so it’s possible I’m going to do the guy a disservice here, but my initial impression.
As people have stated, it’s quite possible the guy is lying. Scientists do falsify results, which is why repetition of controversial results by other completely independent researchers is very important.
A few of parts of the study in particular worry me, firstly is the unspectacular nature of the results before he divides the cohort into fairly arbitrary stimulus seeking groups. I’m a bit worried by his statement of goals (he indicates that the study aims to find an effect) and throughout the study he makes rather weak explanations for any weak results. In other words the guy is pretty blatantly wanting to find something (This is hardly an unusual failing amongst scientists, but it is an important factor to make note of).
I’d also be incredibly wary about the paragraph about how people might not be able to reproduce his results, it’s a pretty stupid thing to put in to start with, but it also suggests to me that he’s had trouble repeating results that he’s not talking about, it’s not like we wouldn’t have expected to have been actively trying to reproduce this for all he’s worth.
95% p value is the absolute minimum to be honest, it’s the one most commonly used but you would also be expected to be able to reproduce those effects in separate experiments.
It’s easy for a researcher to fail to make small consistent mistakes that produce a minor effect and then not realise this.
There’s not really anything I can see here that means that you could completely dismiss it out of hand but it’s not exactly great material (maybe the journal is having a Halloween issue :)). For what it’s worth my money is on undeclared repetition (I’ve met more than one scientist who seem to think this is fine as long as the experiment worked the first time).
You just can’t take research like this seriously until someone independent has repeated the heck out of it, bad research is very common, proof of the paranormal has proved rather elusive.
Having read the article, this is a load of bunk. That is coming from a paranormal enthusiast. These “tests” are stupid, and amount to “interesting brain tricks”. All I’m seeing is that the brain is making valid links of association regardless of the order. That doesn’t really surprise me at all. We have an enormous potential for pattern recognition.
Yep, if I had to bet money on it that would be my first explanation. Although I like the wayward signal from the thalamus explanation too. As I said, they were just goofing around—I think my dad said they were just calibrating stuff in preparation for something they were going to do the following day. I still think it would be fun to mess around with this and see exactly what was going on, though.
I think the thing that makes this interesting is that JPSP, which is about the single most respected journal in social psychology, actually agreed to publish this. The reason, undoubtedly, is that the first author, Daryl Bem, is a really big name in the field – he’s a household word to anyone working in social cognition, due to his development of self-perception theory. I would guess that the editors will include some commentary in the issue in which the article appears, explaining their decision.