NeoAcademic

In what I can only assume is a special issue of Organizational Research Methods, several researchers discuss common statistical and methodological myths and urban legends (MUL) commonly seen in the organizational sciences (for more introduction, see the first article in the series). Fourth and final in this series: Cortina and Landis1 write “The Earth is Not Round (p = .00)”

I had initially hoped to report on this fourth article in the same week as Part 3 of this series, but quickly realized that I would need to parse it with a much finer-tooth comb. Cortina and Landis, with this article, are jumping squarely into the null hypothesis significance testing (NHST) vs. effect size testing (EST) debate with a reasonably strong position. If you aren’t familiar with what these terms refers to, here’s a little reminder:

• NHST: The comparison of obtained differences/relationships to a theoretical sampling distribution to determine the probability that we would find that difference/relationship (or one larger) if there were really no difference/relationship in the population (called the null hypothesis).  If an observed result is improbable (assuming the null hypothesis were true), we typically use the term “statistically significant.”
• EST: The simple reporting of the observed result, such as the size of a correlation or d-statistic, and the confidence with which have made that estimation.

An influential 1994 paper by Cohen2 entitled “The earth is round (p<.05)” raised a large number of valid criticisms of the current state of NHST in psychology, and its strong perspective can be best summarized with this statement:

NHST has not only failed to support the advance of psychology as a science, but also has seriously impeded it. (p. 997)

NHST, Cohen argued, is so commonly misused and misunderstood by psychologists that it had a negative net effect on scientific progress.  I’ve detailed potential problems with NHST elsewhere, so I won’t go into them here, but here’s the basic problem: it’s so easy to simplify statistical significance testing into a simple “it’s significant and therefore a real effect”/”it’s not significant and therefore not a real effect” that many researchers do exactly that, despite the fact that it is a completely invalid conclusion.

Cohen argues that the replacement of NHST with EST would do much good to fix this problem.  Instead of statements like this…

• NHST: The difference between conditions was statistically significant [t(145) = 4.12, p < .05].

…you would see statements like this…

• EST: The difference between conditions was 0.12 standard deviations in magnitude [d = .12, CI(.95): .01 < d < .23].

Same phenomenon; different reporting.  EST discourages researchers from making categorical statements like “there was an effect” while simultaneously giving information about the precision of the estimate obtained.

In the present article, Cortina and Landis argue that this shift will do no good.  They say that while NHST is frequently misused, there at least exists a well-structured system by which to make judgments about NHST.  An effect either is statistically significant, or it is not.  EST, on the other hand, has very few standards by which to make judgments.

The most prominent standard used currently in EST is in fact a bastardization of Cohen’s own article.  As an example, Cohen defined what might be considered “small,” “medium,” and “large” effects, but recommended that researchers come up with their own standards of comparison within individuals research literatures.  And yet, researchers typically use the specific values supplied by Cohen and brandish them as a hammer, making claims about “medium effects” regardless of context.  This is the same “dichotomous thinking” that plagues the use of NHST (although perhaps in this context, it is better called “trichotomous” or “polytomous thinking”).

Cortina and Landis thus argue that a shift to EST will be even worse than with NHST due to the lack of structured expectations with EST.  Although researchers misuse NHST, at least there are standards by which to say there is misuse!  This is a straw man.  Any new technique that gains popularity in the literature will require a “breaking in” period as it is explored independently by new researchers – just look at meta-analysis, SEM, HLM, and other relatively new approaches to data.  A lack of clarity and direction with a new technique does not mean that the old technique is better or even safer.

So what exactly is the urban legend here?  It is that a shift to EST will magically solve the interpretability problems associated with a research literature filled with NHST.  I certainly agree that this belief is a myth and potentially a problem.   Both NHST and EST are simplifications and summaries of much more complex phenomena, and as a result, information is lost along the way.  It takes an expert in the content area, the methodology, and the stats to make valid conclusions about constructs from data.  Sometimes these are the same person; sometimes they are not.  Even with a complete shift to EST, many researchers would continue to overestimate their abilities and misuse the tools they are given.

The authors ultimately conclude that it’s time to simply embrace that taking one analytic approach to a dataset is not enough.  Some combination of well-used NHST and EST is needed.  What I find peculiar, though, is that after decrying a lack of structure and expectations in EST, they give a set of “situations…we as a field should be able to agree upon” that are a little vague:

1. “If one’s sample is very large, then chance is irrelevant.”  An example of N = 50000 is given as an example of a “large sample”.
2. “If one’s sample is very small, then quantitative analysis as a whole is irrelevant.”  This seems ripe for articles citing this and saying “We only had N = 30, so we don’t need to do quantitative analysis.”
3. “If one’s model is simple, then expectations for rejection of chance and for effect magnitudeshould be higher than if one’s model is complex.”
4. “If one’s results could be characterized as counterintuitive, either because of the design or because of the questions being asked, then expectations for magnitude should be lower than for results that replicate previous research or might be characterized as reflecting conventional wisdom.”

Something tells me this problem will not be solved any time soon.

1. Cortina, J., & Landis, R. (2010). The Earth is not round (p = .00) Organizational Research Methods, 14 (2), 332-349 DOI: 10.1177/1094428110391542 [↩]
2. Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49 (12), 997-1003 DOI: 10.1037//0003-066X.49.12.997 [↩]

In what I can only assume is a special issue of Organizational Research Methods, several researchers discuss common statistical and methodological myths and urban legends (MUL) commonly seen in the organizational sciences (for more introduction, see the first article in the series). Third up: Aguinis et al.1 write “Debunking Myths and Urban Legends About Meta-Analysis.”

Meta-analysis has become such a de facto method by which to synthesize a research literature in the organizational sciences that I hardly imagine a modern narrative literature review without one.  If you aren’t familiar with it, meta-analysis essentially involves the computation of a grand mean something across research studies.  This might be a mean difference (usually a Cohen’s d) or a correlation (usually a Pearson’s r).

Unfortunately, the surge in popularity of this statistical technique has brought with it a large number of researchers employing it without really understanding it – imagine the person who computes an ANOVA without any clue what “ratio of the between- to within-group variability” means.  And even if we were to assume all researchers do understand it completely, we now have a large population of “consumers of meta-analyses” that need that same understanding just to accurately interpret a literature review.

Aguinis et al. provide a list of what they believe to be the 7 most common myths and urban legends associated with meta-analysis.  My understanding is that this list came out of a session I attended at SIOP 2010 and subsequent discussions.  I’ll list each of the myths as Aguinis et al. listed them, and my own interpretation of them:

• MUL #1: A single effect size can summarize a literature. Much as you cannot use a sample mean or sample correlation to conclude anything about a single person within that sample, you cannot generalize from a single meta-analytic estimate about any particular setting.  This is why we have moderator analyses; the overall effect size from a meta-analysis only tells you what happens “on average.”  There is not necessarily even a single study or setting where you would find the relationship described by that overall effect size.
• MUL #2: Meta-analysis can make lemonade out of lemons; meta-analysis allows researchers to gather a group of inconclusive and perhaps poorly designed studies and draw impressive conclusions with confidence. Larger samples are certainly gathered by meta-analysis than is possible in a single study, which is certainly a strength of approaching data from this perspective.  But this has led to the common misconception that you can throw anything you want into a meta-analysis and get out “good” results.  It reminds me of the old computer science expression, GIGO: garbage in, garbage out.  If you include only poor quality studies, you’ll get a poor quality average.
• MUL #3: File drawer analysis is a valid indicator of possible publication bias. One of  the techniques recommended to identify if your research suffers from a publication bias (published studies tend to show stronger results than unpublished ones) is to compute a failsafe N.  This value represents how many studies with null results would need to be added to nullify the results of the present meta-analysis.  While a low failsafe N indicates potential publication bias, a high failsafe N does not necessarily indicate the absence of it.
• MUL #4: Meta-analysis provides evidence about causal relationships. GIGO all over again.  If you aren’t meta-analyzing experiments that provide evidence of causality, your meta-analysis will not magically add that interpretation.
• MUL #5: Meta-analysis has sufficient statistical power to detect moderating effects. It’s a common assumption that by meta-analyzing a research literature, you automatically have sufficient power to detect moderators.  While it is true that meta-analyses have greater power to detect moderators than individual primary studies, you do not automatically have sufficient power to detect anything you want to detect.
• MUL #6: A discrepancy between results of a meta-analysis and randomized controlled trials means that the meta-analysis is defective. While a discrepancy might indicate a poorly designed meta-analysis, this is by no means conclusive.  Some discrepancy is inevitable because a meta-analysis is an average of studies, and those studies will vary randomly.
• MUL #7: Meta-analytic technical refinements lead to important scientific and practical advancements. Most refinements in meta-analytic technique do not dramatically alter computed estimates.  Although you should certainly use the most recent refinements (as they will produce the most accurate estimates), you don’t need to worry too much about forgetting one… although there are certainly a few exceptions to this (my own work on indirect range restriction comes to mind!). The biggest mistake is to redo and attempt to publish a meta-analysis that directly replicates another meta-analysis with only minor changes in approach; the difference between the old and new results will almost never be large enough to justify this unless the meta-analytic k is also dramatically increased.

1. Aguinis, H., Pierce, C., Bosco, F., Dalton, D., & Dalton, C. (2010). Debunking myths and urban legends about meta-analysis. Organizational Research Methods, 14 (2), 306-331 DOI: 10.1177/1094428110375720 [↩]