# Meta-analysis: Powerful tool or junk science?
**URL:** https://boards.straightdope.com/t/meta-analysis-powerful-tool-or-junk-science/120919
**Category:** Great Debates
**Created:** [July 26, 2002, 12:43pm UTC](https://boards.straightdope.com/t/meta-analysis-powerful-tool-or-junk-science/120919 "2002-07-26T12:43:14Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@DSeid](https://boards.straightdope.com/u/DSeid)
#### Post date: [July 26, 2002, 12:43pm UTC](https://boards.straightdope.com/t/meta-analysis-powerful-tool-or-junk-science/120919/1 "2002-07-26T12:43:14Z")
In the second-hand smoke thread the issue of the validity of meta-analysis has been questioned. While I think that questioning the technique is specious within the context of that discussion (more on why in a second) I do have by discomfort with its all pervasiveness as a scientific technique, at least within the medical literature.
It has become _the_ tool for so-called “evidence based medicine”. Do a PubMed search for “meta-analysis” and you’ll get a thousand hits for just this calender year. Guidelines are formulated on the basis of meta-analyses. The recent breast cancer screening debate hinges on a new meta-analysis of past data, as do many others. Recomendations on length of antibiotic treatment for urinary tract infections, drug rehab, and on and on, all rely on the results of meta-analyses.
Is it a valid technique or is it so potentiailly flawed as to be used only with extreme caution in special circumstances?
My own take is that it needs to be used much more cautiously than it is. Mainly because of publication bias. Negative trials are less likely to be published therefore meta-analyses are biased to show positive results. Only in the exceptional sitiuation in which you can reasonably believe that you likely have access to all significant trials performed (such as the environmental smoke data, IMHO) is meta-analysis likely to be free of such bias.
Thoughts?
---
### Author:  [@g8rguy](https://boards.straightdope.com/u/g8rguy)
#### Post date: [July 26, 2002, 3:02pm UTC](https://boards.straightdope.com/t/meta-analysis-powerful-tool-or-junk-science/120919/2 "2002-07-26T15:02:26Z")
There’s certainly nothing inherently wrong with it, as you said, but it seems more prone to the possibility of bias. Something else to consider is that if there’s a flaw in some of the original studies which are used for the meta-analysis, that flaw will propagate. Unfortunately, since the meta-analyzers haven’t done the experiment themselves, they’re less likely to catch such a flaw.
In other words, like any other tool, if used properly and carefully it’s fine and if used improperly and incautiously, it’s not.
---
### Author:  [@december](https://boards.straightdope.com/u/december)
#### Post date: [July 26, 2002, 3:03pm UTC](https://boards.straightdope.com/t/meta-analysis-powerful-tool-or-junk-science/120919/3 "2002-07-26T15:03:29Z")
My wife is a biostatistics prof, so your OP is particularly interesting to me, **DSeid**.
At a recent medical school dinner, I was chatting with a co-author of hers who had done a meta-analysis. He had done an enormous amount of work, in reading a huge number of relevant papers and combining tose that could be combined. I was most impressed.
Getting to the questions you asked:
1. I do not know how often meta-analysis is improperly done done.
2. Policy decisions need to utilize the results of all research in a field, one way or another Meta-analysis is a formal way to do this. Without something like meta-analysis, how would the experts decide whether to recommend a mammogram?
3. I agree with you about the potential pitfalls. I suspect that they can be overcome in most cases, with appropriate care.
---
### Author:  [@Cervaise](https://boards.straightdope.com/u/Cervaise)
#### Post date: [July 26, 2002, 8:55pm UTC](https://boards.straightdope.com/t/meta-analysis-powerful-tool-or-junk-science/120919/4 "2002-07-26T20:55:55Z")
My wife is an epidemiologist and works with meta-analytical methods all the time. I’m going to bring this thread to her attention.
---
### Author:  [@Liberal](https://boards.straightdope.com/u/Liberal)
#### Post date: [July 26, 2002, 9:06pm UTC](https://boards.straightdope.com/t/meta-analysis-powerful-tool-or-junk-science/120919/5 "2002-07-26T21:06:22Z")
This promises to be a really great thread. Thanks for opening it, DSeid!
---
### Author:  [@Pythagoras](https://boards.straightdope.com/u/Pythagoras)
#### Post date: [July 26, 2002, 9:12pm UTC](https://boards.straightdope.com/t/meta-analysis-powerful-tool-or-junk-science/120919/6 "2002-07-26T21:12:42Z")
There’s nothing inherintly wrong with it, you just have to make sure you car integrate all the studies statistical data.
---
### Author:  [@ultrafilter](https://boards.straightdope.com/u/ultrafilter)
#### Post date: [July 26, 2002, 9:25pm UTC](https://boards.straightdope.com/t/meta-analysis-powerful-tool-or-junk-science/120919/7 "2002-07-26T21:25:17Z")
The results you get from meta-analysis, as with anything else, are worth no more than the data you start with.
---
### Author:  [@Urban\_Ranger](https://boards.straightdope.com/u/Urban_Ranger)
#### Post date: [July 27, 2002, 7:38am UTC](https://boards.straightdope.com/t/meta-analysis-powerful-tool-or-junk-science/120919/8 "2002-07-27T07:38:11Z")
> [@](#):
>
> \*Originally posted by DSeid \*
> Negative trials are less likely to be published therefore meta-analyses are biased to show positive results.
I suppose that all have to do with assigning weights to various researches. I myself am not familar with the techniques with meta-analyses but it seems to be similar to data mining.
---
### Author:  [@december](https://boards.straightdope.com/u/december)
#### Post date: [July 27, 2002, 3:25pm UTC](https://boards.straightdope.com/t/meta-analysis-powerful-tool-or-junk-science/120919/9 "2002-07-27T15:25:25Z")
Prof. december made the following points:
– She has no problem with meta-analysis, in general
– There are protocols that are supposed to insure that it’s done right
– The analyzers are supposed to contact researchers in the field and ask for unpublished negative results
– She’s not sure that the above step always takes place
– Users are aware that the results of M-A are somewhat subjective
– M-A is a better way to combine studies than what has been used: “We have 4 studies one way and 1 study the other way, so we’ll believe the 4 studies.”
---
### Author:  [@Measure\_for\_Measure](https://boards.straightdope.com/u/Measure_for_Measure)
#### Post date: [July 28, 2002, 1:21am UTC](https://boards.straightdope.com/t/meta-analysis-powerful-tool-or-junk-science/120919/10 "2002-07-28T01:21:12Z")
If there are any primers on meta-analysis on the web, I’d like a link.
---
### Author:  [@The\_Ryan](https://boards.straightdope.com/u/The_Ryan)
#### Post date: [July 28, 2002, 6:09pm UTC](https://boards.straightdope.com/t/meta-analysis-powerful-tool-or-junk-science/120919/11 "2002-07-28T18:09:33Z")
From a strict statistical perspective, meta analysis is worthless. But of course, there a lot of situations where no statistically valid examination is feasible, so meta analysis can be better than nothing. But I think that it’s misleading to use statistical terminology to describe meta analysis, or anything else, that doesn’t follow statistical rules.