How Will #MTurkGate Affect the Validity of Research?

If you haven’t heard by now, #MTurkGate refers to a sudden up-to-400% fee increase by Amazon on MTurk Requesters hiring MTurk Workers to do work on Amazon Mechanical Turk. Or more specifically, MTurkGate refers to the online backlash resulting from that fee increase.

There are a few odd features of the MTurkGate fee increase. Most notably for academic researchers, the price hike is largest for people doing a high volume of small requests – for example, 200 people completing a 30-minute survey. This has, understandably, led many academic researchers to believe that Amazon is targeting them, perhaps with the mistaken impression that social scientists have lots of money to spare. Some researchers have even gone to Facebook and Twitter to demand an academic pricing tier.

Most of the commentary to this point, predictably, has focused on the increased cost. A few have discussed the potential decrease in already-low Worker pay given tight budgets. Undoubtedly, using MTurk will now be more expensive. But no commentary that I have seen has covered the impact on the the validity of research conducted on it. Will this help, harm, or have no impact on the quality of research conclusions using MTurk samples?

The short answer is: it depends.

I’ve written on the use of MTurk by researchers before, and in that discussion, I noted that MTurk is an acceptable source of data for only some research questions – specifically, those research questions where the specific advantages and disadvantages of MTurk don’t harm the conclusions of the particular study you’re trying to do. For example, it’s probably not generally a good idea to conduct research on MTurk where you rely on a surprise, naive reaction, because many MTurk users are completing a huge number of studies and likely to have seen your novel stimulus several times before. If you’re testing something stable and unchanging – like personality or attitudes – then this is less of a problem.

Whether this price hike will change that is an interesting question. In my view, several things could happen here:

• MTurk Requesters cannot afford to run on MTurk anymore, so they stop posting tasks. Fewer tasks potentially means more naive participants. This is potentially a positive for our research conclusions.
• If fewer tasks are posted, veteran MTurk Workers (such as Master Workers) may head elsewhere. Again, this is a surprising positive. If the veterans on MTurk leave for other services, there will be a larger proportion of naive Workers available – although there will potentially be fewer Workers in general. Studies may take a little longer, but that’s not a problem in and of itself.
• If veteran Workers head elsewhere, the popularity of MTurk may follow. This is where things get dicey. If veteran Workers lead a charge to a different, competing website with better pricing, those naive Workers may leave too. Fewer Workers in general means that the population of MTurk becomes highly specialized, which is a distinct problem for researchers. Right now, the characteristics of MTurk are pretty broad – people with all sorts of jobs, backgrounds, expertises, and so on. It is a melting pot of random people. If specific people are encouraged to leave, that diversity could be lost, and if that diversity is important to your research questions, you should look elsewhere for data.
• If the highest performing Workers leave MTurk, remaining Workers may be more desperate for work. Those Workers that stick around may not be the Workers you want. With an even lower pay rate – and MTurk is already quite low – we may end up with a pool of desperate people, quite unlike any other population (in a statistical sense) in whom we might be interested.
• If the proportion of experienced Workers to inexperienced Workers doesn’t change, we probably have nothing to worry about. If experienced and inexperienced Workers leave in equal proportions, we effectively maintain the status quo. Data collection may go a bit more slowly if fewer people are on the service in general, but our results will not be biased by MTurkGate – at least, not any more than they already are.

Importantly, none of these are issues with crowdsourced data in general. They are problems with MTurk specifically. Remember for any research question to carefully match your data collection strategy to your research question. If another crowdsourced data source would be better, use it. If it would be better to abandon crowdsourcing altogether – remember that you have that option. There are even research questions for which Psychology department undergraduates are more generalizable, more predictable, and in general preferable to MTurk Workers. But that’s a question you need to evaluate for yourself, for your own research questions. In convenience sampling, there are no easy answers.