Skip to content

Effective I-O Psychologist and Technologist Collaborations

2019 January 2
tags:
by Richard N. Landers
Why can’t we all just get along?

How can technologists get along with I-O psychologists in the field? This is one of the most common questions I get when consulting. Typically, it starts with something like, “IO: Every time we ask for something to be changed, it seems like IT drags its feet and complains,” or alternatively, “IT: Every time the I-Os ask for something, it shows they don’t understand what they are asking for. At all.” Everyone gets frustrated and development projects seem to take triple the time they should.

So why does this happen?

The key is recognizing that psychologists have a very different base working model of the world than IT experts do. Because of their training, IT experts tend to think in terms of systems – there are inputs, processes, and outputs – and fixing problems means modifying those systems or designing new ones by changing either the data coming in, what’s done with it, or how it is presented.

Psychologists do not think this way. Psychologists instead tend to view systems as fixed entities. For example, few psychologists think of Excel as multiple layers of input, calculation, and data display/visualization subsystems working in tandem. To them, it’s just a “spreadsheet app.”

That ultimately means that psychologists typically do not really appreciate the complexities of system design or inter-system communication. My favorite example of this is in web design from my own experience – a non-systems-thinker (not an I-O in this case, but close enough) was using a web-based survey platform created by a company, but the thinker didn’t like where the logo was placed and asked it to be moved about an inch to the right.

The non-systems thinker did not at all understand why that was a huge and complicated request, yet it was. You see, under the hood, the display system had been built in such a way that the rest of the page was aligned to that image’s placement. You might ask, “Well why would they have done that?,” but as any IT person knows, the code you’ve got is the code you’ve got. Perhaps it should have been written differently the first time, but it’s far too late to change that now. At this point, a lot of things need to be reworked to get that image moved, and all that time is expensive.

To a person with an IT background, the risk of this happening is way more obvious. When another IT person asks, “Can you move the image?,” what they are really asking is, “Do you know if the system has been created in such a way that moving that image will be a small or large project?” To your average I-O psychologist, the website is a static object – just push the image over, what’s the big deal?

This sort of difference in perspective also works the other way. Psychologists tend to conceptualize humans as hugely complicated input-process-output machines but often don’t interconnect those systems. For example, a classic psychological principle is the “mere exposure effect” which describes how familiarity with something generally leads to a preference toward that thing. For example, if you purchase a clock and your roommate/significant other doesn’t particularly like that clock, over time, after seeing it every day, they will like the clock more.

A classic study on mere exposure was about people; it found that the people living at the bottom of the stairwell in an apartment building tended to have more friends in the building than anyone else. They attributed to this to the fact that more people literally passed in front of their door and thus had greater opportunity to say hello. More hellos = more exposure = more friends. The mere exposure effect.

This sort of “fact” about human nature, plus hundreds or thousands of other such facts, form a sort of catalog of effects in a psychologist’s mind: “if you do this to a person, they will be more likely to do that” or “people like this tend to behave that way.”

Some of these “rules” are very specific, which a lot of IT folks don’t really appreciate. So they will just slap together some requested software prototype, assuming it’ll be worked out in testing, making a bunch of little decisions that they see as unimportant, and the psychologists will have a myriad of seemingly random requests, some of which are much more difficult to change than others.

I have found that this sort of misalignment is at the core of problems in IT-psychologist cross-functional teams. The best way to address it is training.

The IT folks need to learn that humans are very complicated and to consult the psychologists on anything human-related in the systems they are building, down to pretty fine details. You need to anticipate the sort of things psychologists might request and build flexibility in assuming they will want changes later. Don’t build to spec; build to possibility (this is also at the core of agile development, but you need to be more careful about it here).

The psychologists need to learn that the systems they are messing with are very complicated and to assume that any small change they request is likely hugely more complex under the hood. Don’t become frustrated when things you think are small can’t be changed; instead, look for workarounds, keep an open mind, and remember that two technical requests with the same outcome are still different requests. “Move the image 1 inch to the right” is a different request from “Add 1 inch of whitespace to the left of the image so that it looks like the image is 1 inch to the right.” One is much easier than the other.

If you work on a tech-IO cross-functional team, someone needs the expertise to identify these sorts of translation issues when they come up. Otherwise, partnerships become dysfunctional. Everyone gets frustrated, assuming the other side is purposefully being difficult.

So, the takeaway? If you are developing technology and I-Os are involved, get an I-O with some IT expertise, an IT expert with some I-O expertise, or preferably, (at least) one of each!

TNTLAB at SIOP 2019

2018 December 3
by Richard N. Landers

It appears that TNTLAB’s move to the University of Minnesota is not slowing us down, as we’ve had another banner-year in terms of SIOP activity, with 13 distinct events for SIOP 2019! That ties our previous record (of 13) two years ago.  If you’re interested in following TNTLAB events at SIOP, here is a convenient list. Come see us!

Dr. Landers presenting…

  1. Panel discussion on the future of assessment technology (Saturday 11:30AM National Harbor 2-3): Sheets, T., Omori, C. L., Andrews, L., Kim, B. H., Landers, R. N., & Mracek, D. L. (2019, April). Predicting prediction: A discussion of technology in assessment & selection.  Panel presented at the 34th Annual Conference of the Society for Industrial and Organizational Psychology, National Harbor, MD.
  2. Rapid-fire IGNITE presentations on new assessment methods (Thursday 5:00PM Chesapeake 4-6): Johnson, T. K., Ellison, L., Ennen, N., Landers, R. N., Prager, R. Y., Oelbaum, Y., Miller, B. H. (2019, April). Candidate experiences matter: Navigating new frontiers in assessment technologies. IGNITE presented at the 34th Annual Conference of the Society for Industrial and Organizational Psychology, National Harbor, MD.
  3. Practical advice on how to anticipate and prevent problems communicating results of analyses (Thursday 10:30AM Potomac D): Chakrabarti, M., Landers, R. N., Tarulli, B. A., & Zabel, K. L. (2019, April). SIOP Select: Communicating results in a complex world. Invited panel at the 34th Annual Conference of the Society for Industrial and Organizational Psychology, National Harbor, MD.
  4. Community of interest exploring new approaches to data presentation and visualization (Friday 11:30AM National Harbor 1): Jankovic, R. & Landers, R. N. (2019, April). Statistics plus data visualizations: How technology brings power to the people. Invited community of interest at the 34th Annual Conference of the Society for Industrial and Organizational Psychology, National Harbor, MD.
  5. Practical skill building workshop session on social media (registration required for CE credit; Saturday 8:00AM Maryland A): Vaughn, D. & Landers, R. N. (2019, April). Friday Seminar 1: Fad to fixture: Social media in the workplace. Invited Friday Seminar at the 34th Annual Conference of the Society for Industrial and Organizational Psychology, National Harbor, MD.
  6. New research methodologies coming out of data science and the training you need to keep up (registered Doctoral Consortium attendees only)

Current or recently graduated students presenting…

  1. Cutting edge in the analysis of text for selection (Saturday 12:30PM Potomac D): Armstrong, M. B. & Landers, R. N. (2019, April). Essay word counts as reflections of general cognitive ability and broad abilities. In Johnson, T. K. M. (Chair) & Gray, R. (Co-Chair), Use Your Words: Text Analysis in Selection and Assessment.  Symposium presented at the 34th Annual Conference of the Society for Industrial and Organizational Psychology, National Harbor, MD.
  2. Using machine learning to get useful prediction and/or construct measurement from trace data (Saturday 8:00AM Maryland A): Auer, E. M., Marin, S., Landers, R. N., Collmus, A. B., Armstrong, M. B., Mujcic, S., & Blaik, J. A. (2019, April). Predicting g with Trace Data: Evidence from a Game-based Assessment.  In J. F. Capman (Chair), Looking Under the Hood: Making Use of Trace Data.  Symposium presented at the 34th Annual Conference of the Society for Industrial and Organizational Psychology, National Harbor, MD.
  3. Back-to-back with the next presentation on this list, a skill-building workshop on creating reproducible and interactive analyses in R and Python (free, no registration required; Thursday 3:30PM Maryland D): Auer, E. M. & Landers, R. N. (2019, April). Creating reproducible and interactive analyses with JupyterLab and Binder. Master tutorial presented at the 34th Annual Conference of the Society for Industrial and Organizational Psychology, National Harbor, MD.
  4. Big skill-building workshop on data science in I-O psychology (free, no registration required; Thursday 1:30PM Maryland 4-6): Auer, E. M., Collmus, A. B., Marin, S., Callan, R. C., & Landers, R. N. (2019, April). Leveraging data science to facilitate insightful, reproducible, and trustworthy I-O. Master tutorial presented at the 34th Annual Conference of the Society for Industrial and Organizational Psychology, National Harbor, MD.
  5. Get up to date on both games and gamification research (Friday 5:00PM Chesapeake J-L): Marin, S. & Landers, R. N. (2019, April). Using gamification’s extrinsic motivators to foster intrinsic motivation. In S. Marin (Chair) & Armstrong, M. B. (Co-Chair), Leveling on Game-Thinking: Research Trends in Gamification and Game-Based Assessments.  Symposium presented at the 34th Annual Conference of the Society for Industrial and Organizational Psychology, National Harbor, MD.

Featuring work by lab members…

  1. A broad exploration of how new technologies are changing classic I-O psychology approaches to problems (and introducing new problems to solve; Friday 8:00AM National Harbor 10-11): McCloy, R. A., Sinclair, A., Koch, A. J., Purl, J. D., Dalal, R. S., Oswald, F. L., & Landers, R. N. (2019, April). Measurement: New methods for classic problems, classic methods for new problems.  In R. A. McCloy & Kell, H. J. (Chair), New/Classic Methods for Classic/New Problems.  Symposium presented at the 34th Annual Conference of the Society for Industrial and Organizational Psychology, Presented National Harbor, MD.
  2. Symposium exploring technology-mediated interivews and all the new problems they introduce (Saturday 8:00AM Chesapeake A-C): White, J. C., Auer, E. M. & Behrend, T. S. (2019, April). The influence of applicant accent and resume in technology-mediated interviews. In M. Langer and K. G. Melchers (Chairs),  Job Interview Technology: Effects on Applicants, Evaluators, and Adverse Impact. Symposium presented at the 34th Annual Conference of the Society for Industrial and Organizational Psychology, National Harbor, MD.

TNTLAB Has Moved to the University of Minnesota

2018 October 5
by Richard N. Landers

UMN LogoIt has been a long while since I’ve posted on this blog, because, well, it’s been a busy summer. Let me give you the short of it: TNTLAB has moved to the University of Minnesota.

My last post was April 27, which was about a week after the Spring 2018 SIOP conference in Chicago. That timing is not coincidental, as it was only about a week before that conference that I accepted the job offer to move from Old Dominion University (ODU) to the University of Minnesota. It has been a bit of a whirlwind since then, and I feel like I should get my feelings on paper (or equivalent). So this post will be a little more journal-ish than usual (please forgive my self-indulgence) and also quite long. I hope though that it will capture some of the complex feelings and issues associated with moving from one tenure-track I/O position to another, which is something for which I found no resources when I tried to hunt for them this summer. It is a bit like your first academic job, except that you a bit better understand what you’re bringing on top of yourself.

Some History

It’s a bit weird to be at Minnesota, because I actually graduated from Minnesota with a Ph.D. in I/O psychology in 2009, and have thus been away for 9 years. When I was here then, the consistent faculty were Paul Sackett (my adviser), John P Campbell, Deniz Ones, and Joyce Bono. Steve Motowidlo was there when I started but left after a year or two, taking one student in my cohort with him.  Nathan Kuncel started toward the end. So that is the Minnesota I remember.

Old Dominion University

After I graduated, I started a tenure-track job at Old Dominion University in I/O with Debbie Major, Karin Orvis, and Don Davis. Thus, I’ve spent my entire time away from Minnesota as faculty at ODU, where I achieved tenure a few years ago. In that time, I saw (and helped with) some significant transformations in that program; at the end, Debbie had become an Associate Dean, Karin had left, Don had retired, and Konstantin Cirgularov, Xiaoxiao Hu, and Violet Xu had joined, making me the most senior faculty member in the area as of Fall 2017. There was clearly an expectation that I was moving into an “area head” sort of role as of the time I left, which I still feel a significant degree of guilt about.

Why I Had Considered Moving

I had always been a hair dissatisfied with my position at ODU, not because of my colleagues, but because of the climate and resources of the university and our college. ODU’s psychology program, like Georgia Tech’s and a handful of others, is housed in a College of Sciences instead of a College of Liberal Arts, which brings with it the values of the natural sciences. That means different things in different universities. When I interviewed with the (since-let-go-for-related-reasons) dean of the college in 2009, he mentioned that Oceanography had recently bought a $1 million submarine on a grant and asked how I planned to compete with that, a tall order for an I-O. Thus, I spent a now-regrettable amount of time thinking about how to make ODU more money instead of how to build my own research program. It gave me a lot of experience with the NSF and the grant-earning process – which is useful, to be sure – but it was also quite overwhelming, having had no grant training whatsoever while I was a graduate student.

This is not to say that ODU is a bad place to work. Instead, I say it to emphasize the impact of office politics on the career trajectory of an academic. This sort of thing happens everywhere. That dean’s priority was grants; there was no ambiguity in this. If you did not have the potential for grants, or a grant record, you could be denied, discouraged, etc. in regards to career advancement, i.e., tenure, and that shapes the sort of research you find yourself doing. As I mentioned earlier, that dean has since been removed from his position, so things are a bit different now. But it still shaped my “growing up,” so to speak, and had left a persistent case of “that grass is probably greener” that I found difficult to shake.

Dean Wormer

At least the dean never put the lab on double-secret probation (Animal House).

As a result, I had applied to and interviewed for other jobs on and off throughout my time there. It wasn’t entirely due to this niggling dissatisfaction though; this is also related to one of those open secrets in academia; the only way to get a decent raise, or more resources, or really anything else you might want is to get an external job offer. As a pre-tenure faculty member (and even afterward), it is frankly unwise not to try to get another job, however disloyal and uncomfortable that makes you feel. This is especially salient to I/Os given research on lifetime earning potential related to your salary at your first job; that first job sets a salary trajectory that tends to follow you for the rest of your life. Missing a $10K salary bump by forcing your university to compete with an external job offer can result in millions of dollars of lifetime earnings lost. The only way to escape it in academia is to get two job offers simultaneously. So it’s best to act early.

Unfortunately (I thought at the time), none of those jobs worked out for one reason or another. I interviewed at a few business schools and found the cultures insufferable: an incredible amount of attention paid to the “caliber” of journal people publish in as measured by impact factors. Most business schools even had “the list” of journals that it was acceptable to publish in. This entire idea makes me bristle. It is essentially all of the worst aspects of goal setting; you encourage people to optimize their behavior not on what is actually good for them personally, good for their university, good for society, etc., but what is good for getting a high impact factor, i.e., incremental growth of well-established theories. It also directly encourages the factors that led to the replication crisis, and more subtly, it encourages people to research “safe” topics that they know will get them published in those journals. It diminishes the role of creativity in social science. As a consequence, it is a major reason why most current OB research as published in places like AMJ or JOM is essentially useless to any real live human being working in management. The whole approach is simply against my core values; so I stopped looking too seriously at business schools. Importantly, there are some specific OB programs that do not fall in this trap, but I find they are pretty rare.

I also interviewed at a few I/O programs, with varying levels of success, but often had the same problem: philosophical disagreements. One of the things that is not very obvious as a graduate student is that individual graduate programs have ingrained cultures that are both shaped by faculty behavior and shape that behavior. It is a self-promoting cycle of behavior, and it crosses area lines. So just like ODU had a “grants-first” attitude across the College of Sciences because of the dean at that time, I encountered programs that were basically business-lite; all theory, all the time, with very little attention paid to the real world. I didn’t like that much either, because it seems directly contrary to what I see as I/O’s core values.

The Invitation

Moonshot

Good luck with that.

When I was invited to apply to the Minnesota job, I was immediately interested. The faculty here still create theory, but that theory is in service to a particular problem that needs solving. I was also immediately skeptical, because I didn’t think I had a snowball’s chance in hell of getting the job. Applying felt like a moonshot. I even emailed Paul and asked if he was sure I was supposed to have been on the email invitation list and got an answer I wasn’t expecting: yes, because they were looking for complementary skill sets to those already in the area. They did not want another Paul, Deniz, Nathan, or Aaron Schmidt (who had joined after I graduated). They wanted something new.

At this point, let me give you a bit of additional self-disclosure that might clarify my skepticism. I am an interdisciplinary researcher. I try to blend fields, mostly I-O with human-computer interaction or computer science, and create something new. That means, quite frankly, that it is more difficult to publish in the traditional “top tier” outlets of any of the fields I integrate. The reasons for this are complicated, and not worth going into too much detail here, but the core of it is (again) value differences. The aspects of research that computer scientists find “top tier” worthy are not the same as the ones that I/Os do. A lot of what I/O does in its top tier outlets seems completely inconsequential to a computer scientist; when I talk about I/O research to that crowd, I hear a lot of “why would anyone care about that?” and “can’t the problem be fixed with different software?”  The reverse is also true.

That means when publishing this sort of research, you must make a choice: do I publish in a well-respected interdisciplinary journal that people in the two component fields probably won’t read or do I try to tailor it to one of the fields at the expense of the other? This is not a meta-question most I/Os need to ask themselves. I have generally tended toward interdisciplinary journals and have done pretty well with it. But the result is that I have most of a reputation within the particular interdisciplinary communities I publish in (primarily games and gamification) and less so within the I/O community itself, at least as far as publishing goes. This also has some strong “impact factor” implications, which I don’t care about personally but many people do. So let me just say I was surprised when Paul indicated that difference in approach, perspective, and skill set, despite these downsides in terms of “program stats,” is actually what they were looking for.

John P. Campbell

The man himself, judging both of us.

The specific then-open position was created as a result of John P. Campbell’s retirement. If you’re an I/O, you know who this is. If you think of job performance as a behavior and differentiate it from effectiveness, that is thanks to John. If you think of performance as the results of declarative knowledge, procedural knowledge and and skill, and motivation, that is thanks to John. If you think of performance as something you can break down into eight categories of behavior, that is thanks to John. His accomplishments are significant and varied, and this just scratches the surface. He is a giant in the field.

So as a result of John’s retirement, the remaining faculty at Minnesota solicited donations to create an endowed chair: the John P. Campbell Distinguished Professorship in Industrial and Organizational Psychology. After an invitation to apply, a job interview, and several tense weeks of negotiation, that is the job I now hold. It is frankly a bit intimidating to follow in such footsteps. But where I think John and I share a philosophy it is in our willingness to challenge the status quo when we believe something with conviction. Just as John challenged people to rethink what job performance is at a time when people felt just peachy about supervisory ratings, I contend that the ways I/O tends to think about the role and importance of technology in workplace behavior is very wrong, that it betrays some fundamental misunderstandings of what technology is, why it exists, and how it is created by, interfaces with, and affects humans in turn.

The Move

Since agreeing to take this position, I spent the summer finding a house and moving and am still in the midst of moving into my research lab. The most recent issue is that the flooring in what will become my main lab as it turns out is full of asbestos! It’s being removed this week in fact. This is only the most recent in a long string of challenges in setting up a new lab from the ashes of an old one, but that’s another (long) post, perhaps for later.

I was fortunate to be able to bring two students with me: Elena Auer and Sebastian Marin.  Elena had taken over as my lab manager at ODU, so her being able to come to Minnesota is a bit of a godsend. If you get no advice out of this post except one nugget, it’s this one: don’t move your lab alone, if you can avoid it. Having (at least) a second person creates a sense of continuity that will keep you grounded. Otherwise, it’s easy to lose yourself in the new university system. But do be keenly aware that whereas you made the choice to upend your own life and research to move, you are inflicting this choice upon your graduate students. Even if the move is better for you, it’s not always better for them. To try to even the balance a little, I built in a few years of GRA funding into my startup budget for both of them; and frankly, it’s the least I could do.

The sheer scope difference in terms of complexity has been particularly salient for me. Minnesota is a traditional “R1.” ODU was trying to move in that direction, but it was not there yet when I left. And honestly, I didn’t realize how far it had to go until moving here, where the sheer scope of internal resources and staff available for research support is a bit staggering.  For example, 110,000 hours of computing time on a supercomputer is the “free tier” of service here per lab per year; but if you give them a paragraph justifying why you need it, they’ll give you 440,000 hours! (For reference, entire research projects for us tend to use less than 500).  The disparity in resources also makes a lot of the comments we received on NSF applications over the years make a lot more sense, e.g., “I am not sure this university has the resources to support a project like this.”  Incidentally, it does, but it definitely would’ve required a lot more work on my part than if we’d done it at Minnesota.

Thus, the change is ultimately very much to our advantage even if we are trapped in moving hell right now.  In terms of some specific gains, one, we get to use that supercomputer for our artificial intelligence work.  ODU had one too, but it was smaller in scope, and as a result, not particularly social-scientist-friendly, which is not at all true here.  Two, there is a respect for research in administration that makes resource allocation much less of an issue relatively-speaking; there seems to always people or units or other systems to meet particular needs.  Three, and most excitingly to me, I am building a whole-room VR lab to play with! VR is still in its infancy, but some key areas of advantage versus traditional approaches have begun to take shape in the selection and training domains that I want to test here. And as an added benefit, it’s a ton of fun.

Conclusion

TNTLABTNTLAB is still mid-transition, but we’re getting closer every day back to “normal operations” every day. I think we will be more productive than ever before; the lab has already grown larger people-wise than it was at ODU. I will chronicle some of this on the blog so that you can see the growth yourself, if you are curious enough to indulge me again!