In the second-hand smoke thread the issue of the validity of meta-analysis has been questioned. While I think that questioning the technique is specious within the context of that discussion (more on why in a second) I do have by discomfort with its all pervasiveness as a scientific technique, at least within the medical literature.
It has become the tool for so-called “evidence based medicine”. Do a PubMed search for “meta-analysis” and you’ll get a thousand hits for just this calender year. Guidelines are formulated on the basis of meta-analyses. The recent breast cancer screening debate hinges on a new meta-analysis of past data, as do many others. Recomendations on length of antibiotic treatment for urinary tract infections, drug rehab, and on and on, all rely on the results of meta-analyses.
Is it a valid technique or is it so potentiailly flawed as to be used only with extreme caution in special circumstances?
My own take is that it needs to be used much more cautiously than it is. Mainly because of publication bias. Negative trials are less likely to be published therefore meta-analyses are biased to show positive results. Only in the exceptional sitiuation in which you can reasonably believe that you likely have access to all significant trials performed (such as the environmental smoke data, IMHO) is meta-analysis likely to be free of such bias.
There’s certainly nothing inherently wrong with it, as you said, but it seems more prone to the possibility of bias. Something else to consider is that if there’s a flaw in some of the original studies which are used for the meta-analysis, that flaw will propagate. Unfortunately, since the meta-analyzers haven’t done the experiment themselves, they’re less likely to catch such a flaw.
In other words, like any other tool, if used properly and carefully it’s fine and if used improperly and incautiously, it’s not.
My wife is a biostatistics prof, so your OP is particularly interesting to me, DSeid.
At a recent medical school dinner, I was chatting with a co-author of hers who had done a meta-analysis. He had done an enormous amount of work, in reading a huge number of relevant papers and combining tose that could be combined. I was most impressed.
Getting to the questions you asked:
I do not know how often meta-analysis is improperly done done.
Policy decisions need to utilize the results of all research in a field, one way or another Meta-analysis is a formal way to do this. Without something like meta-analysis, how would the experts decide whether to recommend a mammogram?
I agree with you about the potential pitfalls. I suspect that they can be overcome in most cases, with appropriate care.
I suppose that all have to do with assigning weights to various researches. I myself am not familar with the techniques with meta-analyses but it seems to be similar to data mining.
– She has no problem with meta-analysis, in general
– There are protocols that are supposed to insure that it’s done right
– The analyzers are supposed to contact researchers in the field and ask for unpublished negative results
– She’s not sure that the above step always takes place
– Users are aware that the results of M-A are somewhat subjective
– M-A is a better way to combine studies than what has been used: “We have 4 studies one way and 1 study the other way, so we’ll believe the 4 studies.”
From a strict statistical perspective, meta analysis is worthless. But of course, there a lot of situations where no statistically valid examination is feasible, so meta analysis can be better than nothing. But I think that it’s misleading to use statistical terminology to describe meta analysis, or anything else, that doesn’t follow statistical rules.