I’m not sure that this is a weakness in the study given the result. Since no effect was found does it really make a difference how much prayer was or was not received by any patient? After all if x = 0 the a*x also = 0 for all a.
I’m inclined to think not. What do those who are expert in the design of experiments think?
It’s pretty unlikely that they’d find no effect whatsoever. I’m thinking that they found no statistically significant effect and the journalist(s) either didn’t understand that or dumbed it down for the article.
It’s a pretty strong objection, though–if you’re trying to measure the effect of different values of a variable on health and you can’t control for that variable, what’s the point?
I guess maybe I see what you are saying. But if there is a small, random fluctuation around zero as a result of noise then multiplying this by a constant is still without meaning because it is probably noise.
So wouldn’t “no statistically significant effect” multiplied by a factor still be “no statistically significant effect”?
This would only matter if you were trying to contol for an absence of prayers. If ten people praying fails to cure somebody of cancer, it doesn’t matter if 100 other people were praying secretly as well. That doesn’t make a zero into a one.
I guess that small fluctuation is really indeterminite. We have no way of knowing whether it is a real effect or is merely noise (experimental error or other indeterminacy) or a combination, so we really can’t say anything about it other than “no statistical significance.”
And, yes, that would seem to mean that the study isn’t very strong evidence either way.
On the other hand, it looks like a losing proposition for nay sayers. A study that showed an effect would be claimed as being strengthened by adding an indeterminate amount of additional prayer. Even though, it seems to me, if an uncontrolled variable weakens one study it should weaken all.
If the groups were large enough, and it was randomly determined which was the control group (so that there was no reason to suspect any particular difference between the groups except for which group they were in), then the amount of extra prayer received by each group would, on the average, be the same. Things would balance out that way. (The extent to which they might/might not balance out exactly is taken into account by statistical theory, and depends on things like sample size.)
This is a factor accounted for in statistical studies in general. There are often all sorts of other factors, in addition to the one being studied, that can affect the results, which is why you can’t establish for sure whether an effect can be attributed to the thing you’re studying and not other chance factors. But with proper sampling techniques, you can assume those other factors would be the same for all groups and would tend to “average out.” A “statistically significant” difference is one that is probably not accounted for by those other factors (because it would be highly improbable to find such a big difference if all else was, in fact, equal).
No, you have to control for the amount of prayer (or aspirin, or drug X, or…) that every subject receives. What you’re saying would imply that whether a prayer study is well-designed depends on the effect it finds, and that’s not a reasonable position.
Only if the relationship is linear. (well not only). But tere can easily be a “model” whereby there is no significant effect until you reach a critical level of prayer. I may puch on a big rock, and not move it. Ten poeple may push on a big rock and not moveit. But a hundred people could.
Good point. The control group would be expected to receive the same amount of unknown prayer as the other two groups. Ergo (I think maybe) the experiment would be measuring the difference between the intentionally administered prayer and the unknown amount of prayer each group received. And there was no statistically significant difference.
I disagree, If the effect is always zero then it doesn’t really matter if the “amount” is controlled for. That would suggest that “too much” prayer would prevent it from being effective. If you definitely know that some of the subjects were prayed for and you definitely know that none of those subjects were affected by it then I don’t see how unaccounted “extra” prayer matters unless you want to suggest that the extra prayer prevented it from being effective.
Diogenes is right. Think about it this way: If the studies showed that prayer did effect cures in some people but not others, then it would make sense to see if the cured subjects were receiving extra prayers. Since the study found no effect, extra prayers are irrelevant.