3 responses

  1. Fletcher Christensen
    February 21, 2014

    The two big questions I’d have relative to this are: (1) what are the results like on job performance as opposed to academic performance and (2) do the extant studies account for what I would consider probably the most important factor with respect to academics—the status of the recommender within her/his field?

    Obviously, (2) is going to be less of an issue at the admissions level, but in domains where letters play a significant role in hiring (like academia), it seems like it might be rather important. Off the top of my head, one could look at tenure attainment as an outcome with predictors like publication count, log citation count, years since degree, and possibly some measure of service as potential predictors.

    In any case, thanks for the blog post. It does seem like an interesting question.

    • Richard N. Landers
      February 23, 2014

      The relationship with actual job performance was my first question – unfortunately, that literature is even less developed than this one.

      The role of individual differences in letter-writing is pretty massive – aside from expertise effects, there are even personality effects. The issue here is really one of reliability – on average, across 3 letters, do you end up tapping a reasonable amount of true score potential?

      I don’t know that academic outcomes at the level you’re talking about would be all that useful. In many academic fields (including I/O Psych), most PhDs don’t even go into academia. Among those that do, you’re going to end up with severe range restriction. Perhaps a binary indicator of a job offer in the field of training within a year after graduation would be preferable?

  2. Jorn Tygood
    January 18, 2015

    I think it would be very difficult to quantify letters of recommendation. The post mentions the common selection biases; that means breaking it down to a counting issue is questionable. But there are other difficulties when you want to judge the quality of a letter somehow. For instance, not everyone is a talented writer and sometimes, what was supposed to be a great recommendation, just sounds like a mediocre reference. That’s why tools such as [link] have become so popular in recent years. And furthermore, how do you take into account the “quality” of the referee? Certainly there is a difference between the Junior Lecturer and the Dean of the Business School; and a reader will pick up on that. But how do you rate that in numbers?

Leave a Reply




Back to top
mobile desktop