NeoAcademic

In his 2008 book, Outliers, Malcolm Gladwell criticizes the traditional “top-down” selection method used by organizations hiring on the basis of cognitive ability (colloquially referred to as “intelligence”).  By his argument, there is a certain level at which additional cognitive ability simply is not valuable.

It is certainly an attractive idea; Gladwell argues that it is historical advantages that bias scores on employee selection devices.  John is lucky to come from a good family, happens to put himself in a position to take advantage of the opportunities that present themselves, and gets hired to a solid position more by luck than by skill.  By this logic, there are many qualified applicants; some were simply more lucky than others, and hiring from the top-down merely punishes the unlucky ones without any real benefit to the organization.  And if the success of your highest performing employees is due to luck, there’s not much reason to worry about selecting the best of the best, right?

In a recent article in Psychological Science, Arneson, Sackett, and Beatty1 call Gladwell’s approach the good-enough hypothesis while they call the traditional approach the more-is-better hypothesis.  They test the relative value of these hypotheses by comparing cognitive ability and performance in four large datasets containing a total of 170,286 people:

• Project A.  In the late 1980s, US soldiers took part in a large scale project designed to link various selection devices to soldier performance, which included the ASVAB, which itself contained a strong cognitive ability component.  Actual ratings of job performance were standardized and compared with cognitive ability scores derived from the ASVAB.
• College Board SAT.  This dataset contained SAT scores and GPA data.  GPA was standardized within schools and compared with SAT performance.
• Project TALENT.  The Department of Health, Education and Welfare (before Health and Education split in the late 1970s) ran a longitudinal study high school seniors, starting in 1960, tracking a large group of them through their college GPA.  This GPA was compared with a composite ability score collected from the study.
• National Education Longitudinal Study of the Class of 1988.  Standardized verbal and mathematics ability scores of eighth graders were compared with their later college GPAs.

To determine which hypothesis better described the data across these datasets, the authors did the following:

1. Checked for ceiling effects (to ensure that the data toward the upper end of ability was not clustered)
2. Used a lowess smoother on the scatterplot of the ability-performance relationship to visualize the relationship at all levels of ability
3. Used a power polynomial test to identify curvilinearity in the relationship
4. Used a second power polynomial test on only the high ability folks (z > 1)
5. Compared regression slopes (b) of performance on ability at four segments of the relationship: minimum to z = -1, -1 < z < 0, 0 < z < 1, and 1 to maximum

Across analysis types, it became clear that the more-is-better hypothesis better described the data.  But more surprisingly, ability appeared to be more important at the high end than it was at the low end.  Or in other words, the relationship between ability and performance gets stronger among those with high ability.  Gladwell could not have been more inaccurate in describing these relationships.

Lowess curves from Arneson, Sackett and Beatty (2011)

Interestingly, Gladwell may have previously had a more anti-selection bent in his book that was perhaps edited down later after he became aware of the massive research literature on hiring using psychological measurement.

1. Arneson, J., Sackett, P., & Beatty, A. (2011). Ability-Performance Relationships in Education and Employment Settings: Critical Tests of the More-Is-Better and the Good-Enough Hypotheses Psychological Science DOI: 10.1177/0956797611417004 [↩]