Skip to content

Stats and Methods Urban Legend 3: Myths About Meta-Analysis

2011 May 3

ResearchBlogging.orgIn what I can only assume is a special issue of Organizational Research Methods, several researchers discuss common statistical and methodological myths and urban legends (MUL) commonly seen in the organizational sciences (for more introduction, see the first article in the series). Third up: Aguinis et al.1 write “Debunking Myths and Urban Legends About Meta-Analysis.”

Meta-analysis has become such a de facto method by which to synthesize a research literature in the organizational sciences that I hardly imagine a modern narrative literature review without one.  If you aren’t familiar with it, meta-analysis essentially involves the computation of a grand mean something across research studies.  This might be a mean difference (usually a Cohen’s d) or a correlation (usually a Pearson’s r).

Unfortunately, the surge in popularity of this statistical technique has brought with it a large number of researchers employing it without really understanding it – imagine the person who computes an ANOVA without any clue what “ratio of the between- to within-group variability” means.  And even if we were to assume all researchers do understand it completely, we now have a large population of “consumers of meta-analyses” that need that same understanding just to accurately interpret a literature review.

Aguinis et al. provide a list of what they believe to be the 7 most common myths and urban legends associated with meta-analysis.  My understanding is that this list came out of a session I attended at SIOP 2010 and subsequent discussions.  I’ll list each of the myths as Aguinis et al. listed them, and my own interpretation of them:

  • MUL #1: A single effect size can summarize a literature. Much as you cannot use a sample mean or sample correlation to conclude anything about a single person within that sample, you cannot generalize from a single meta-analytic estimate about any particular setting.  This is why we have moderator analyses; the overall effect size from a meta-analysis only tells you what happens “on average.”  There is not necessarily even a single study or setting where you would find the relationship described by that overall effect size.
  • MUL #2: Meta-analysis can make lemonade out of lemons; meta-analysis allows researchers to gather a group of inconclusive and perhaps poorly designed studies and draw impressive conclusions with confidence. Larger samples are certainly gathered by meta-analysis than is possible in a single study, which is certainly a strength of approaching data from this perspective.  But this has led to the common misconception that you can throw anything you want into a meta-analysis and get out “good” results.  It reminds me of the old computer science expression, GIGO: garbage in, garbage out.  If you include only poor quality studies, you’ll get a poor quality average.
  • MUL #3: File drawer analysis is a valid indicator of possible publication bias. One of  the techniques recommended to identify if your research suffers from a publication bias (published studies tend to show stronger results than unpublished ones) is to compute a failsafe N.  This value represents how many studies with null results would need to be added to nullify the results of the present meta-analysis.  While a low failsafe N indicates potential publication bias, a high failsafe N does not necessarily indicate the absence of it.
  • MUL #4: Meta-analysis provides evidence about causal relationships. GIGO all over again.  If you aren’t meta-analyzing experiments that provide evidence of causality, your meta-analysis will not magically add that interpretation.
  • MUL #5: Meta-analysis has sufficient statistical power to detect moderating effects. It’s a common assumption that by meta-analyzing a research literature, you automatically have sufficient power to detect moderators.  While it is true that meta-analyses have greater power to detect moderators than individual primary studies, you do not automatically have sufficient power to detect anything you want to detect.
  • MUL #6: A discrepancy between results of a meta-analysis and randomized controlled trials means that the meta-analysis is defective. While a discrepancy might indicate a poorly designed meta-analysis, this is by no means conclusive.  Some discrepancy is inevitable because a meta-analysis is an average of studies, and those studies will vary randomly.
  • MUL #7: Meta-analytic technical refinements lead to important scientific and practical advancements. Most refinements in meta-analytic technique do not dramatically alter computed estimates.  Although you should certainly use the most recent refinements (as they will produce the most accurate estimates), you don’t need to worry too much about forgetting one… although there are certainly a few exceptions to this (my own work on indirect range restriction comes to mind!). The biggest mistake is to redo and attempt to publish a meta-analysis that directly replicates another meta-analysis with only minor changes in approach; the difference between the old and new results will almost never be large enough to justify this unless the meta-analytic k is also dramatically increased.
  1. Aguinis, H., Pierce, C., Bosco, F., Dalton, D., & Dalton, C. (2010). Debunking myths and urban legends about meta-analysis. Organizational Research Methods, 14 (2), 306-331 DOI: 10.1177/1094428110375720 []
Previous Post:
Next Post:
3 Responses leave one →
  1. May 3, 2011

    I like the point about reproducible research. It would be good if more meta-analyses included the complete database of studies and accompanying effect sizes, sample sizes, and study properties in a form ready for re-analysis. It would be even better if the meta-analytic analyses were embodied in some reproducible statistical document.

  2. May 3, 2011

    I like the point about reproducible research. It would be good if more meta-analyses included the complete database of studies and accompanying effect sizes, sample sizes, and study properties in a form ready for re-analysis. It would be even better if the meta-analyses were produced using some form of reproducible statistical document like Sweave.

    Meta-analyses could be seen as an ongoing collective endeavour, where alterations to methodology could be easily examined by future researchers.

  3. May 12, 2011

    While that would be fantastic, I think the biggest roadblock is that the current academic atmosphere does not support such a thing. Releasing data from a meta-analysis is akin to sharing data – and even if that data has been sliced and diced, there’s always a voice in the back of your head that says “maybe there’s another publication in that data.” Given the current reward system in academic, I don’t see this happening any time soon (even if it would be great move for science).

Leave a Reply

Note: You can use basic XHTML in your comments. Your email address will never be published.

Subscribe to this comment feed via RSS