Question for statisticians about new prayer study.

Maybe this should be in GQ but I think a debate could easily ensue so I’ll start it here.

And …

Yes, I know there is a thread about this already. However, I have a different question. This story in the Los Angeles Times contains a criticism of the study that may or may not be valid.

I’m not sure that this is a weakness in the study given the result. Since no effect was found does it really make a difference how much prayer was or was not received by any patient? After all if x = 0 the a*x also = 0 for all a.

I’m inclined to think not. What do those who are expert in the design of experiments think?

It’s pretty unlikely that they’d find no effect whatsoever. I’m thinking that they found no statistically significant effect and the journalist(s) either didn’t understand that or dumbed it down for the article.

It’s a pretty strong objection, though–if you’re trying to measure the effect of different values of a variable on health and you can’t control for that variable, what’s the point?

I guess maybe I see what you are saying. But if there is a small, random fluctuation around zero as a result of noise then multiplying this by a constant is still without meaning because it is probably noise.

So wouldn’t “no statistically significant effect” multiplied by a factor still be “no statistically significant effect”?

I think you’re begging the question there. That x=0 is the conclusion of the study, not a premise.

In this case the test statistic is probably the difference in recovery rates. If that’s small enough, no significant effect is observed, but if it gets larger that’ll change.

In this case the test statistic is probably the difference in recovery rates. If that’s small enough, no significant effect is observed, but if it gets larger that’ll change.

I agree the objection makes no sense. If the study concluded that prayer has no effect, then it makes no difference whether every prayer was accounted for. It wouldn’t change the results.

But the unaccounted prayers wouldn’t make it larger.

Not exactly. x = 0 is the null hypothesis. The conclusion of the study is that there’s not enough evidence to reject the idea that x = 0.

True. It’s possible that the guy was objecting to prayer studies in general and the newspaper misidentified his criticism as specific to this one.

This would only matter if you were trying to contol for an absence of prayers. If ten people praying fails to cure somebody of cancer, it doesn’t matter if 100 other people were praying secretly as well. That doesn’t make a zero into a one.

I guess the objection is that if prayers received for patient cannot be controlled, you can’t prove that the amounts of prayer weren’t roughly equal.

If there’s the possiblity that everyone received equal amounts of prayers, then the fact that there was no difference in patient recovery rates does not argue against prayer having an effect.

I guess that small fluctuation is really indeterminite. We have no way of knowing whether it is a real effect or is merely noise (experimental error or other indeterminacy) or a combination, so we really can’t say anything about it other than “no statistical significance.”

And, yes, that would seem to mean that the study isn’t very strong evidence either way.

On the other hand, it looks like a losing proposition for nay sayers. A study that showed an effect would be claimed as being strengthened by adding an indeterminate amount of additional prayer. Even though, it seems to me, if an uncontrolled variable weakens one study it should weaken all.

If the groups were large enough, and it was randomly determined which was the control group (so that there was no reason to suspect any particular difference between the groups except for which group they were in), then the amount of extra prayer received by each group would, on the average, be the same. Things would balance out that way. (The extent to which they might/might not balance out exactly is taken into account by statistical theory, and depends on things like sample size.)

This is a factor accounted for in statistical studies in general. There are often all sorts of other factors, in addition to the one being studied, that can affect the results, which is why you can’t establish for sure whether an effect can be attributed to the thing you’re studying and not other chance factors. But with proper sampling techniques, you can assume those other factors would be the same for all groups and would tend to “average out.” A “statistically significant” difference is one that is probably not accounted for by those other factors (because it would be highly improbable to find such a big difference if all else was, in fact, equal).

No, you have to control for the amount of prayer (or aspirin, or drug X, or…) that every subject receives. What you’re saying would imply that whether a prayer study is well-designed depends on the effect it finds, and that’s not a reasonable position.

Only if the relationship is linear. (well not only). But tere can easily be a “model” whereby there is no significant effect until you reach a critical level of prayer. I may puch on a big rock, and not move it. Ten poeple may push on a big rock and not moveit. But a hundred people could.

Good point. The control group would be expected to receive the same amount of unknown prayer as the other two groups. Ergo (I think maybe) the experiment would be measuring the difference between the intentionally administered prayer and the unknown amount of prayer each group received. And there was no statistically significant difference.

I disagree, If the effect is always zero then it doesn’t really matter if the “amount” is controlled for. That would suggest that “too much” prayer would prevent it from being effective. If you definitely know that some of the subjects were prayed for and you definitely know that none of those subjects were affected by it then I don’t see how unaccounted “extra” prayer matters unless you want to suggest that the extra prayer prevented it from being effective.

But the rock didn’t move. Postulating unknown rock pushers is pointless if the rock didn’t move. It wouldn’t explain a lack of effect.

Diogenes is right. Think about it this way: If the studies showed that prayer did effect cures in some people but not others, then it would make sense to see if the cured subjects were receiving extra prayers. Since the study found no effect, extra prayers are irrelevant.

How did I know that this question would generate a statistically signigicant amount of debate, you ask?

Being a tyro in the subject I lean both ways. I’m currently leaning toward the view of Boink (do you mind if I use just the last name?) and DtC by 2:1 at a confidence level of 95%.

I think the idea that all of the groups on average received the same amount of unsolicited prayer is persuasive.