Evaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research
Open Access
- 13 March 2013
- journal article
- research article
- Published by Public Library of Science (PLoS) in PLOS ONE
- Vol. 8 (3), e57410
- https://doi.org/10.1371/journal.pone.0057410
Abstract
Amazon Mechanical Turk (AMT) is an online crowdsourcing service where anonymous online workers complete web-based tasks for small sums of money. The service has attracted attention from experimental psychologists interested in gathering human subject data more efficiently. However, relative to traditional laboratory studies, many aspects of the testing environment are not under the experimenter's control. In this paper, we attempt to empirically evaluate the fidelity of the AMT system for use in cognitive behavioral experiments. These types of experiment differ from simple surveys in that they require multiple trials, sustained attention from participants, comprehension of complex instructions, and millisecond accuracy for response recording and stimulus presentation. We replicate a diverse body of tasks from experimental psychology including the Stroop, Switching, Flanker, Simon, Posner Cuing, attentional blink, subliminal priming, and category learning tasks using participants recruited using AMT. While most of replications were qualitatively successful and validated the approach of collecting data anonymously online using a web-browser, others revealed disparity between laboratory results and online results. A number of important lessons were encountered in the process of conducting these replications that should be of value to other researchers.Keywords
This publication has 47 references indexed in Scilit:
- Economic Games on the Internet: The Effect of $1 StakesPLOS ONE, 2012
- Conflict adaptation in time: Foreperiods as contextual cues for attentional adjustmentPsychonomic Bulletin & Review, 2011
- Cooperation and Contagion in Web-Based, Networked Public Goods ExperimentsPLOS ONE, 2011
- Amazon's Mechanical TurkPerspectives on Psychological Science, 2011
- Using Amazon Mechanical Turk for linguistic researchPsihologija, 2010
- Task Switching Across the Life Span: Effects of Age on General and Specific Switch Costs.Developmental Psychology, 2005
- Stroop-type interference: Congruity effects in color naming with typewritten responses.Journal of Experimental Psychology: Human Perception and Performance, 1998
- Effects of masked stimuli on motor activation: Behavioral and electrophysiological evidence.Journal of Experimental Psychology: Human Perception and Performance, 1998
- The attentional blinkTrends in Cognitive Sciences, 1997
- The flankers task and response competition: A useful tool for investigating a variety of cognitive problemsVisual Cognition, 1995