Should You Hire BlazinWeedClown@Mail.Com?

In recent article by Blackhurst, Congemi, Meyer and Schau1 in The Industrial-Organizational Psychologist, e-mail addresses from a group of 14,718 people who had applied for entry-level jobs in manufacturing were examined for their appropriateness.  The researchers found that roughly 25% of e-mail addresses were inappropriate or antisocial, and that the level of inappropriateness predicted several qualities of interest to hiring managers: conscientiousness, professionalism, and work-related experience.  Interestingly, cognitive ability was not related.

The types of e-mail addresses found appear in the table below, which were extracted by a team of 25 graduate students with high inter-rater reliability (a sub-sample of 1000 was used for this purpose).

From SIOP's TIP webpage

The graduate students next categorized all 15,000ish e-mail addresses (600 addresses assigned to each of the 25 students).  At the same time, the graduate student coded the e-mails as either “appropriate when applying to a job,” “questionable,” or “inappropriate when applying to a job.”  Afterward, one of the researchers reviewed all 15,000, brought any questionable judgments to the attention of a 3-person panel for discussion.  The researchers then compared mean scores on cognitive ability, conscientiousness, professionalism, and work-related experience across those with appropriate, questionable, and inappropriate e-mail addresses.  Statistically significant differences were found on all dimensions.

Unfortunately, statistical significance is easy to attain in this sample.  Even tiny effects will be statistically significant.  The article did not report any standard deviations to give us a sense of effect sizes, so I had to do a little detective work.  Here’s a table comparing outcomes for specific subtypes of inappropriate e-mail addresses:

From SIOP's TIP webpage

This is the only table containing means, and fortunately, there are also degrees of freedom – that means we can reverse-engineer the t-test formula to estimate the standard deviation of this scale.  It’s not perfect, but it’s the best available.  Because these are independent-samples t-tests, t equals the mean difference (in this case, code group minus control group) divided by the pooled standard error (roughly s/SQRT(N)), and we can get the value of N by adding 2 to df (some of the numbers are a little odd in here – for example, DF should be equal to N*2-2, but it’s not – so this is my best guess).  If we assume the SD of each group is equal, we can use the following formula to solve for s: s = sqrt(N)*(mean difference)/T.  That produces SDs for each group here between 54.6 and 58.9, so if we assume these SDs hold up for the other scales, the differences on the predictors between appropriate and inappropriate e-mail addresses range from d=0.00 to d=0.11.  So these are not by any stretch of the imagination big effects.  But they are effects, about in line with what we’d expect from the intercorrelations between psychological predictors and performance generally.

The study is not without limitations; all of the measures provided were by a consulting firm, so we do not have any way to independently verify their content.  The e-mail address ratings were also made by graduate students, and it is not clear how well their judgments would generalize to actual hiring managers. Actual hiring decisions made later were not available, nor was job performance data, so validation evidence is missing.  All we really have is a new correlate of predictors.

Interestingly, the study also identified that roughly 5% of e-mail addresses contained information that looked like a date; considering legislation forbidding discrimination on the basis of age, the legality of hiring managers having access to this information is unclear.  Although e-mail address appropriateness predicts characteristics of interest (and thus should potentially be included in a packet of information used for hiring), it may contain information itself inappropriate for a manager to see (and thus should not be included).  Further research is needed to explore this further.

1. Blackhurst, E., Congemi, P., Meyer, J. & Sachau, D. (2011). Should you hire BlazinWeedClown@Mail.Com? The Industrial/Organizational Psychologist, 49(2), 27-38. [↩]