Stats and Methods Urban Legend 3: Myths About Meta-Analysis

In what I can only assume is a special issue of Organizational Research Methods, several researchers discuss common statistical and methodological myths and urban legends (MUL) commonly seen in the organizational sciences (for more introduction, see the first article in the series). Third up: Aguinis et al.1 write “Debunking Myths and Urban Legends About Meta-Analysis.”

Meta-analysis has become such a de facto method by which to synthesize a research literature in the organizational sciences that I hardly imagine a modern narrative literature review without one.  If you aren’t familiar with it, meta-analysis essentially involves the computation of a grand mean something across research studies.  This might be a mean difference (usually a Cohen’s d) or a correlation (usually a Pearson’s r).

Unfortunately, the surge in popularity of this statistical technique has brought with it a large number of researchers employing it without really understanding it – imagine the person who computes an ANOVA without any clue what “ratio of the between- to within-group variability” means.  And even if we were to assume all researchers do understand it completely, we now have a large population of “consumers of meta-analyses” that need that same understanding just to accurately interpret a literature review.

Aguinis et al. provide a list of what they believe to be the 7 most common myths and urban legends associated with meta-analysis.  My understanding is that this list came out of a session I attended at SIOP 2010 and subsequent discussions.  I’ll list each of the myths as Aguinis et al. listed them, and my own interpretation of them:

• MUL #1: A single effect size can summarize a literature. Much as you cannot use a sample mean or sample correlation to conclude anything about a single person within that sample, you cannot generalize from a single meta-analytic estimate about any particular setting.  This is why we have moderator analyses; the overall effect size from a meta-analysis only tells you what happens “on average.”  There is not necessarily even a single study or setting where you would find the relationship described by that overall effect size.
• MUL #2: Meta-analysis can make lemonade out of lemons; meta-analysis allows researchers to gather a group of inconclusive and perhaps poorly designed studies and draw impressive conclusions with confidence. Larger samples are certainly gathered by meta-analysis than is possible in a single study, which is certainly a strength of approaching data from this perspective.  But this has led to the common misconception that you can throw anything you want into a meta-analysis and get out “good” results.  It reminds me of the old computer science expression, GIGO: garbage in, garbage out.  If you include only poor quality studies, you’ll get a poor quality average.
• MUL #3: File drawer analysis is a valid indicator of possible publication bias. One of  the techniques recommended to identify if your research suffers from a publication bias (published studies tend to show stronger results than unpublished ones) is to compute a failsafe N.  This value represents how many studies with null results would need to be added to nullify the results of the present meta-analysis.  While a low failsafe N indicates potential publication bias, a high failsafe N does not necessarily indicate the absence of it.
• MUL #4: Meta-analysis provides evidence about causal relationships. GIGO all over again.  If you aren’t meta-analyzing experiments that provide evidence of causality, your meta-analysis will not magically add that interpretation.
• MUL #5: Meta-analysis has sufficient statistical power to detect moderating effects. It’s a common assumption that by meta-analyzing a research literature, you automatically have sufficient power to detect moderators.  While it is true that meta-analyses have greater power to detect moderators than individual primary studies, you do not automatically have sufficient power to detect anything you want to detect.
• MUL #6: A discrepancy between results of a meta-analysis and randomized controlled trials means that the meta-analysis is defective. While a discrepancy might indicate a poorly designed meta-analysis, this is by no means conclusive.  Some discrepancy is inevitable because a meta-analysis is an average of studies, and those studies will vary randomly.
• MUL #7: Meta-analytic technical refinements lead to important scientific and practical advancements. Most refinements in meta-analytic technique do not dramatically alter computed estimates.  Although you should certainly use the most recent refinements (as they will produce the most accurate estimates), you don’t need to worry too much about forgetting one… although there are certainly a few exceptions to this (my own work on indirect range restriction comes to mind!). The biggest mistake is to redo and attempt to publish a meta-analysis that directly replicates another meta-analysis with only minor changes in approach; the difference between the old and new results will almost never be large enough to justify this unless the meta-analytic k is also dramatically increased.

1. Aguinis, H., Pierce, C., Bosco, F., Dalton, D., & Dalton, C. (2010). Debunking myths and urban legends about meta-analysis. Organizational Research Methods, 14 (2), 306-331 DOI: 10.1177/1094428110375720 [↩]