If you haven’t heard by now, #MTurkGate refers to a sudden up-to-400% fee increase by Amazon on MTurk Requesters hiring MTurk Workers to do work on Amazon Mechanical Turk. Or more specifically, MTurkGate refers to the online backlash resulting from that fee increase.
There are a few odd features of the MTurkGate fee increase. Most notably for academic researchers, the price hike is largest for people doing a high volume of small requests – for example, 200 people completing a 30-minute survey. This has, understandably, led many academic researchers to believe that Amazon is targeting them, perhaps with the mistaken impression that social scientists have lots of money to spare. Some researchers have even gone to Facebook and Twitter to demand an academic pricing tier.
Most of the commentary to this point, predictably, has focused on the increased cost. A few have discussed the potential decrease in already-low Worker pay given tight budgets. Undoubtedly, using MTurk will now be more expensive. But no commentary that I have seen has covered the impact on the the validity of research conducted on it. Will this help, harm, or have no impact on the quality of research conclusions using MTurk samples?
The short answer is: it depends.
I’ve written on the use of MTurk by researchers before, and in that discussion, I noted that MTurk is an acceptable source of data for only some research questions – specifically, those research questions where the specific advantages and disadvantages of MTurk don’t harm the conclusions of the particular study you’re trying to do. For example, it’s probably not generally a good idea to conduct research on MTurk where you rely on a surprise, naive reaction, because many MTurk users are completing a huge number of studies and likely to have seen your novel stimulus several times before. If you’re testing something stable and unchanging – like personality or attitudes – then this is less of a problem.
Whether this price hike will change that is an interesting question. In my view, several things could happen here:
- MTurk Requesters cannot afford to run on MTurk anymore, so they stop posting tasks. Fewer tasks potentially means more naive participants. This is potentially a positive for our research conclusions.
- If fewer tasks are posted, veteran MTurk Workers (such as Master Workers) may head elsewhere. Again, this is a surprising positive. If the veterans on MTurk leave for other services, there will be a larger proportion of naive Workers available – although there will potentially be fewer Workers in general. Studies may take a little longer, but that’s not a problem in and of itself.
- If veteran Workers head elsewhere, the popularity of MTurk may follow. This is where things get dicey. If veteran Workers lead a charge to a different, competing website with better pricing, those naive Workers may leave too. Fewer Workers in general means that the population of MTurk becomes highly specialized, which is a distinct problem for researchers. Right now, the characteristics of MTurk are pretty broad – people with all sorts of jobs, backgrounds, expertises, and so on. It is a melting pot of random people. If specific people are encouraged to leave, that diversity could be lost, and if that diversity is important to your research questions, you should look elsewhere for data.
- If the highest performing Workers leave MTurk, remaining Workers may be more desperate for work. Those Workers that stick around may not be the Workers you want. With an even lower pay rate – and MTurk is already quite low – we may end up with a pool of desperate people, quite unlike any other population (in a statistical sense) in whom we might be interested.
- If the proportion of experienced Workers to inexperienced Workers doesn’t change, we probably have nothing to worry about. If experienced and inexperienced Workers leave in equal proportions, we effectively maintain the status quo. Data collection may go a bit more slowly if fewer people are on the service in general, but our results will not be biased by MTurkGate – at least, not any more than they already are.
Importantly, none of these are issues with crowdsourced data in general. They are problems with MTurk specifically. Remember for any research question to carefully match your data collection strategy to your research question. If another crowdsourced data source would be better, use it. If it would be better to abandon crowdsourcing altogether – remember that you have that option. There are even research questions for which Psychology department undergraduates are more generalizable, more predictable, and in general preferable to MTurk Workers. But that’s a question you need to evaluate for yourself, for your own research questions. In convenience sampling, there are no easy answers.
Grad School Series: Applying to Graduate School in Industrial/Organizational Psychology
Starting Sophomore Year: Should I get a Ph.D. or Master’s? | How to Get Research Experience
Starting Junior Year: Preparing for the GRE | Getting Recommendations
Starting Senior Year: Where to Apply | Traditional vs. Online Degrees | Personal Statements
Alternative Path: Managing a Career Change to I/O | Pursuing a PhD Post-Master’s
Interviews/Visits: Preparing for Interviews | Going to Interviews
In Graduate School: What to Expect First Year
Rankings/Listings: PhD Program Rankings | Online Programs Listing
A career in I/O psychology requires a Master’s degree or Ph.D., and most of the resources I’ve presented in my graduate school series are intended for those on the “traditional path” to graduate school. Most PhD students these days, whether in I/O or otherwise, finish their bachelor’s degree and head straight to Master’s or Ph.D. training.
This is certainly the easiest way. As with any career, the earlier you know what you want to do with your life, the easier it is to set yourself down the path to get there. But that doesn’t mean a career change to I/O psychology is impossible. It just mean it will take a little more work.
As I’ve noted elsewhere, the first decision when considering a career in I/O is whether you will enter a Master’s or Ph.D. program. Normally, you would first consider the sort of job you might want: I/O’s with Master’s degrees tend to be the “technicians” of our field, applying I/O knowledge out in the world, whereas I/Os with Ph.D.’s tend to be the “researchers” of our field, conducting research studies within organizations to apply I/O knowledge out in the world but also to build that knowledge. Practically speaking, however, a career change into a Master’s program is substantially easier than a career change into a Ph.D. program.
The reason for this is the type of experience needed to apply for each degree. Research experience is useful for a Master’s application, but it is only necessary for a Ph.D. application. If you’re already out in industry and don’t see research experience as a realistic option for yourself, you might want to start by targeting a Master’s degree. In that path, experience as an HR professional, and preferably in strategic HR, will be most helpful to your application in lieu of research experience.
If you do want to strive for the Ph.D., you need research experience. The easiest way to get it will be to find a local college or university with psychology researchers that you can easily drive to. Use Google to search for specific people doing work you find vaguely interesting at that university, and then email and call those faculty members. Explain that you are interested in getting research experience and are willing to donate 10 to 20 hours per week of your time, preferably at home, but that you are willing to come in for meetings. This will give you the best chances of being taken on as a volunteer research assistant. Remember that you don’t need I/O experience specifically, although this is certainly better if you can get it. Any experience as a research assistant in psychology will help your application.
One of the biggest challenges you will face is taking the GRE. If you haven’t been in school for a while, studying for the GRE will probably bring back a lot of bad memories. But it’s worth it; I/O psychologists helped design and validate the GRE, so we take it seriously. You should too. If you haven’t studied in a while – potentially years – give yourself plenty of time to prep. I recommend starting at least a year in advance.
Finally, your personal statement is critical. This is where you explain why you want to change careers. Remember that taking you on as a graduate student is a risk for your new advisor too. That person is worried that 1) you don’t really know what kind of commitment you’re getting into and 2) you heard that I/O was a high paying, fast growing field, and that’s the only reason you applied. Remember that a Master’s program will be at least a 40 hour per week commitment, and a PhD program will be 60 to 80. You need to explain, convincingly, that you understand that and undertake the challenge knowingly and willingly. Remember that even online programs require this sort of time commitment.
Once you have all of your preparation and materials in order, the next big challenge in a career change is figuring out where to apply. My most important advice: don’t be lured into applying to a program because 1) the application requirements are easy, 2) the timeline matches yours, 3) they accept credits for “life experience,” 4) it’s cheap, or 5) it’s an online for-profit. All of these are signals that the program is not going to lead you to a new job. They are generally intended for people who already have a job lined up, but someone said, “for promotion, you need a higher degree.” If you don’t already have a job lined up, don’t go into those sorts of programs.
This continues my thoughts on SIOP 2015 about Big Data. In that article, I described how I/O psychologists can meaningfully position themselves as the “meaning-makers” of Big Data. But what I worry about is that I/O psychologists will not take the opportunity to become proactive in regards to innovative technology.
To date, we have almost universally been reactive, and as has been common in this situation, a new innovation is signified by a flurry of conference activity followed by near-silence in the academic literature.
To illustrate, I remember when the idea of “maybe social media is important” hit the SIOP conference maybe 3 or 4 years ago. The first year, there were a couple of presentations on social media. The next year, there were at least five or six, several standing-room only. This year, there were two again. I imagine next year it will decrease to one or none. In all these years, the I/O research literature has added maybe two papers on the topic, although a fair amount of research on workplace social media has appeared in non-I/O journals. And – surprise – social media is still a concern for those in the field.
I think what this represents is interest in the practitioner community – specifically, someone has asked what their informed, I/O opinion is on this new technology – yet the research literature remain mostly silent. In the absence of empirical research, the practitioner community develops its own internal understanding of the technology and then stops talking about it at SIOP. Thus the issue remains a concern in the field while few contribute empirical research. Decisions are made based upon rules of thumb and gut reactions.
This situation is not tenable if I/O hopes its research to be applied to the modern workplace. We risk the same pattern followed by newly minted MBAs and teachers – a brain full of knowledge from school that is discarded within the first year of “real” work. If we want I/Os to continue applying research on the organizational front lines, we need to craft more relevant research.
I don’t mean to imply by this that the responsibility lies solely with the academics. We seem to have forgotten the scientist-practitioner model, which does not imply that there are both scientist and practitioners, but rather that all I/O psychologists should be both scientists and practitioners. The harsh realities of billable hours make this balance difficult for people in the field, but more must be done. We must increase academic-practitioner partnerships for the good of our field, to produce research at the forefront of new organizational phenomena.
And perhaps it goes without saying, but if you have or are considering such a technology, give me a call!!