Overemphasis on Theory Development Is Damaging Organizational Psychology
In a recent article appearing in Organizational Psychology Review, Pillutla and Thau1 make some very strongly worded arguments about the role of theory development in psychological science. I’ll start exploring their paper with a quote in their own words:
The state of [industrial/organizational psychology] and its obsession with novel theoretical contributions is antithetical to the goals of the scientific method.
The authors lament the over-reliance on the development of novel theory as the foundation for a meaningful contribution to scientific research. They suggest, rightfully I think, that the pursuit of novel theory creates a perverse incentive structure for scientists, encouraging them to pursue identification of “facts” which may fit available data but do not really exist, ultimately leading to the creation of theories that do not truly complement or build upon any prior theories, orphans of the scientific literature that provide no true value to our understanding of organizational phenomena. They label this approach the pursuit of the “interesting” at the expense of good science.
They further argue that this state of affairs is in fact the result of a fundamental misunderstanding of the scientific process. The enterprise of science is not one where we should aim to identify facts, because facts do not really exist; instead, it is one where theory is used as a tool to incrementally better understand phenomena, each additional bit of research contributing to a better description of a real research problem. When we pursue conclusions like “this is how organizations function” or “this is how employees behave”, we are misleading both ourselves those that rely upon our research.
Certainly, such misunderstandings have occurred before, with highly damaging implications for the progress of science. One such misunderstanding was the over-reliance on statistical significance testing, where “statistically significant” was mistakenly taken as synonymous for “real” or “important”. Although people still make such mistakes in interpretation, the movement toward effect size interpretation and model specification (as evidenced through changing APA guidelines for reporting) is clearly targeted at reducing this continuing problem. In their article, Pillutla and Thau are arguing that theory development as the sole target of science is the research-methods equivalent of the statistical-significance interpretation problem – that this over-emphasis in academic publishing is pervasive and harmful. Like statistical significance testing, sometimes novel theory development is the right approach, but many times it isn’t.
Through analysis of citations to one of the most dominant proponents of theory-building as the basis of all science, the authors also contend that management, business, and applied psychology have been disproportionately harmed by this viewpoint, in comparison to other areas of the social sciences.
This view is a bit extreme, but may be the front edge of a coming wave of reform. I’ve noticed myself over the last few years that fewer and fewer papers appearing in our top journals – Journal of Applied Psychology, Personnel Psychology, Academy of Management Journal, etc. – have solved any real problems for actual organizations. Instead, they focus on exploring and specifying increasingly minute aspects of organizations – interesting, sure – but not terribly useful in any tangible sense. From chatting with practitioners and other early career scholars, I’ve discovered this view is not all that unusual. One person at the Society for Industrial and Organizational Psychology conference this year even told me, “I haven’t seen anything I could actually use to help my employees in Journal of Applied Psychology for at least five years.” That is a terrible state of affairs. If we’re not solving real problems, what value do we really provide to both organizations (whom we study) and taxpayers (who pay us, or at least those of us working in state institutions)? The push to publish only in top tier journals – where such theory development is required – only exacerbates this problem.
Interesting theories are not, by themselves, worthy of investigation unless they are put to the service of explaining research problems… the quest for novelty and interestingness of facts has infused them with significance without any regard to the knowledge that they generate.
So are things as dire as Pillutla and Thau indicate? There have certainly been a number of high-profile cases in psychology lately that have caused the public to question the value of our entire field. And we should absolutely work to repair that damage, but the path to do so is unclear. Some researchers have taken a purely empirical approach, attempting to replicate controversial papers and examine their convergence, eschewing theory entirely in the quest for reliable knowledge. Is that enough? I’m guessing we’ll find out in the next five or ten years.
- Pillutla, M., & Thau, S. (2013). Organizational sciences’ obsession with “that’s interesting!”: Consequences and an alternative Organizational Psychology Review, 3 (2), 187-194 DOI: 10.1177/2041386613479963 [↩]
SIOP 2013 Coverage: Schedule Planning | Day 1 | Day 2 | Day 3
Day 3 was the final day of the SIOP conference. Per tradition, sessions were lightly attended – too much late night enjoyment at the end of Day 2, I suspect! I had a slow start myself, only getting to poster sessions a little after 10AM. One that caught my eye explored the effect of organizational support for the use of technology over time; surprisingly, the effect of strong organizational support on technology use weakened over time, not quite in line with what prior theory (and the researcher presenting!) predicted.
I also decided somewhat last-minute to attend a Master Tutorial on Bayesian statistics for I/O. Unfortunately, despite a full house and CE credits on the line for many, the speaker didn’t show! After about 15 minutes, I left and headed back to the poster session on Research Methods, with lots of interesting work, including a bit more on the equivalence of mTurk and undergraduate research samples. The added value on this paper? English-speaking, US-based mTurk samples are similar to undergraduate samples, but non-English-speaker samples are a little different.
Finally, I ended up at the closing plenary, given by Reverend TJ Martinez, founder of the Cristo Rey Jesuit College. He related an inspiring story about starting with zero support to start a school but ending by graduating a fair number of low-SES high school students (with an abysmal high school completion rate prior) with a 100% entrance rate into college. I am still not sure precisely what he had to do with I/O, but he was certainly an entertaining and admirable speaker.
At the very end incoming SIOP President Tammy Allen laid out a 5-section plan for SIOP in the coming year:
- Tie the varied local I/O advocacy and interest groups better to SIOP.
- Increase the visibility of I/O Psychology in Introductory Psychology courses and textbooks. Graduates of undergraduate psychology programs should, at minimum, be able to answer the question, “What is I/O Psychology?”, and this doesn’t happen in most programs.
- Work to improve SIOP’s branding, which may include the launch of a new practitioner-oriented journal.
- Improve national advocacy efforts on behalf of SIOP. Apparently we have already some folks on retainer, but this will expand.
- Better understand SIOP’s role and position in “science.” Psychology is viewed as one of the foundational areas of scholarship on which many other areas are based, but where does I/O fit specifically?
And that’s it for SIOP 2013! I’ve got another day in Houston – which will likely involve the Houston Aquarium and a nice dinner somewhere – and then back to cooler Norfolk (at a relatively chillier 70 degrees F!), where I’ll be writing a slew of follow-up emails to all the intriguing folks I met this year. And hopefully I’ll be sufficiently recovered from this trip in a year to attend SIOP 2014 in sunny Honolulu!
| 14:04 | #SIOP13 Day 3 begins with another SIOP tradition, for me anyway: I’ve lost my voice! Too much of a good thing I suppose!! |
| 16:21 | Fascinating poster on org support for tech use in orgs – changes over time in unexpected ways #siop13 |
| 16:22 | @ericknud by that logic, wouldn’t you want to be unattractive? Inverted u-curve perhaps? #siop13 |
| 18:22 | @ericknud yes, that’s what I’m saying – it is probably more complicated than just “moderately attractive is good” |
| 18:24 | #siop13 is both amazing and frustrating because you run into people you know constantly… Missed a whole session! |
| 18:26 | @TSP_Consulting @jbrodieg and what do #SIOP points get me, hmm? #siop13 or #siop14 perks perhaps? |
| 18:41 | Intro to Bayesian Statistics speaker is missing 10 mins into session… Uhoh #siop13 |
| 18:46 | #SIOP13 skipped out on Bayesian tutorial without a speaker for research methods poster session… At least they’ll all show up! |
| 18:51 | @jsnread that was the consensus of the room! |
| 18:53 | @BelindaK04 what room is this? |
| 21:55 | #siop13 closing plenary… Rev Martinez is a pretty funny guy! |
| 21:57 | #siop13 closing plenary… Low income students as an investment opportunity for orgs… Future customers, future employees |
| 22:10 | #siop13 closing plenary… Inspiring speaker on the power of the few and disenfranchised to accomplish their goals |
| 22:18 | #siop13 #SIOP Prez Tammy Allen’s goals: tie local IO to SIOP, IO in intro, branding incl practitioner journal, natl advocacy, map of science |
| 22:19 | #siop13 video on Hawaii to encourage attendees at #siop14… Do we really need convincing?? |
| 23:06 | End of Day 3 and end of #siop13… Great conference! So many followup emails to write! |
SIOP 2013 Coverage: Schedule Planning | Day 1 | Day 2 | Day 3
Day 2 of SIOP 2013 was a bit busier for me. After an early breakfast, I headed to my first session of the day at 8:30, a discussion of social media in selection, which ended up really being a discussion of LinkedIn in selection. Like Day 1, this was a bit depressing. Although there was a lot of talk about social media, there was not much data, which is what I was really hoping for. I did hear a few interesting tidbits in this session that I had not heard before. For example, a lot of managers look up job candidates on LinkedIn, where many pieces of information are shared that would not be part of a job application, like profile pictures and affiliated religious groups. Once a hiring manager sees such information, their judgment has potentially been irreparably affected – or more critically, the possibility that it has been so affected creates a substantial legal risk. How can an organization prove that manager wasn’t influenced by the job candidate’s race, skin color, national origin, gender or religion once they have been exposed to it?
The next session on social media was a little better, data-wise, but it wasn’t perfect. One of the presentations provided some data on the criterion-related validity of LinkedIn profiles for predicting job performance, as an extension of a previously published paper doing the same for Facebook profiles. As it turns out, LinkedIn profiles can provide valuable information to hiring managers. But I asked a question of the speakers: Aren’t we just setting ourselves up to redo this same study for every social media platform of the moment? If we seek a scientific understanding of the useful information provided by social media, don’t we need a better way to do it? Everyone seemed to think that was a great idea, but alas, I didn’t get any specific recommendations on how to go about it.
Next, I headed to a symposium on online simulation, which was absolutely packed. Much in line with what I noticed from Day 1, practitioners are hungry for data on how these technologies have and should affect their practice – and yet again, data was a bit light. They presented some interesting assessment tools, but they seemed like relatively small advancements over what we already have – for example, the use of branching video-based simulations rather than branching text or static video assessments. The practitioners did bring up some interesting issues they faced in launching these technologies globally. For example, when they tried to launch their video-based simulation in another country with poor Internet infrastructure, they discovered that bandwidth was not the guarantee that it is in the US – their client couldn’t view the videos at even quasi-decent quality. This resulted in some rather last minute changes to meet client needs.
After this symposium, I headed over to the poster session on training – in an hour, I got through only about 75% of the room. Hopefully I’ll have a bunch of papers incoming to my inbox in the next few days so that I can revisit them in more detail than a 3′ x 5′ poster can provide!
Finally, at 5PM, in a symposium with my colleagues at ICF and ARI on emerging training technologies, I gave my presentation on gamification, which was by a large margin the most successful presentation I have ever given at SIOP in the 9 years I have attended. In it, I presented a theoretical model of gamification providing two specific mechanisms by which gamification can affect learning in the context of education/training, and I tested one of those processes empirically, providing several practical recommendations for those wishing to gamify their own organizational learning. I had literally 25 minutes of questions and comments after the session ended!
All in all, an exceptionally successful Day 2! Day 3 is a bit lighter, and I suspect tonight will be a bit more intense, so Day 3 may also start a bit late. But I am still hopeful for some great learning and some great conversations.
| 13:13 | Hangovers and spotty session attendance… so starts Day 2 of #siop13… Although many of the hangovers won’t start for hours yet! |
| 13:41 | Yet another session on social media for selection… Some data in this one, I hope? #siop13 |
| 14:05 | Information on protected classes may be more common on LinkedIn than in resumes… Greater legal risk (managers cannot unsee!) #siop13 |
| 14:06 | @ChrisWieseIO The speakers seem to be mixing selection predictors and selection methods… Seems a very jumbled approach to me. #siop13 |
| 14:10 | I’m getting a little tired of hearing “maybe we’ll have some empirical research later” in these social media symposia #siop13 |
| 14:26 | @ericknud I’ve got several running right now… Just sad there are not more here #siop13 |
| 14:34 | I am sad when people count the number of stat. significant relationships in their results as an indicator of result stability #siop13 |
| 14:47 | @ericknud Open for now! Stick around, will probably head up front |
| 15:17 | @ericknud was talking to Don until just now, out in the hall now for a bagel |
| 15:17 | Discovered I was hoarse from last night’s activities while asking a question at a symposium… whoops #siop13 |
| 17:12 | Full house, standing room only at innovations in online simulation #siop13 |
| 17:29 | To practitioners, video-based virtual assessment center apparently = long-form video SJTs with open ended responses #siop13 |
| 17:36 | Cultural issues within orgs can be unexpected and tricky for innovative assessment: “our org doesn’t use email” #siop13 |
| 17:55 | Online assessment center design with branching reminds me of game design process #siop13 |
| 19:02 | Did not even get all the way through the training poster session in an hour – too much interesting stuff! #siop13 |
| 19:03 | RT @sandyfias: It makes me sad that about 8/3800 I/O psychologists at #siop13 r tweeting. And we ask how we can be more visible and rele … |
| 19:03 | @sandyfias I’d suggest we have a tweetup, but it would just be depressing! 😉 #siop13 |
| 20:22 | Hope you’re attending my session at #siop13 on serious games and gamification in Room 339AB at 5PM today! (Room change!) |
| 20:25 | @BelindaK04 @PsychoSoAnt @sandyfias Are there enough people? I’ve got 3-4:30 tomorrow #siop13 |
| 21:55 | Starting in 5 mins, session at #siop13 on serious games and gamification in Room 339AB! |
| 22:05 | Standing room only at the session! #siop13 http://t.co/HLZwNOwTpY |
| 22:37 | RT @jsnread: Gamified learning results in longer time spent on task, thus greater learning. Wish I could have gamified my dissertation. … |
| 22:38 | @neilmorelli thanks – got a little rushed at the end but hopefully it made sense! #siop13 |
| 23:27 | Wow – 25 minutes of questions about #gamification research after session! Thanks everyone! #siop13 |
| 23:27 | @jsnread ha, why not!? |
| 23:50 | #siop13 Day 2 is over! Now to the ODU alumni dinner (where I entertain) and the Minnesota alumni event (where I am entertaining) |