Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants
Top Cited Papers
- 12 March 2015
- journal article
- research article
- Published by Springer Science and Business Media LLC in Behavior Research Methods
- Vol. 48 (1), 400-407
- https://doi.org/10.3758/s13428-015-0578-z
Abstract
Participant attentiveness is a concern for many researchers using Amazon’s Mechanical Turk (MTurk). Although studies comparing the attentiveness of participants on MTurk versus traditional subject pool samples have provided mixed support for this concern, attention check questions and other methods of ensuring participant attention have become prolific in MTurk studies. Because MTurk is a population that learns, we hypothesized that MTurkers would be more attentive to instructions than are traditional subject pool samples. In three online studies, participants from MTurk and collegiate populations participated in a task that included a measure of attentiveness to instructions (an instructional manipulation check: IMC). In all studies, MTurkers were more attentive to the instructions than were college students, even on novel IMCs (Studies 2 and 3), and MTurkers showed larger effects in response to a minute text manipulation. These results have implications for the sustainable use of MTurk samples for social science research and for the conclusions drawn from research with MTurk and college subject pool samples.This publication has 16 references indexed in Scilit:
- Distrust and the positive test heuristic: Dispositional and situated social distrust improves performance on the Wason Rule Discovery Task.Journal of Experimental Psychology: General, 2014
- Investigating Variation in ReplicabilitySocial Psychology, 2014
- Reputation as a sufficient condition for data quality on Amazon Mechanical TurkBehavior Research Methods, 2013
- Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchersBehavior Research Methods, 2013
- Evaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral ResearchPLOS ONE, 2013
- Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical TurkPolitical Analysis, 2012
- Conducting behavioral research on Amazon’s Mechanical TurkBehavior Research Methods, 2011
- Instructional manipulation checks: Detecting satisficing to increase statistical powerJournal of Experimental Social Psychology, 2009
- Encoding Under Trust and Distrust: The Spontaneous Activation of Incongruent Cognitions.Journal of Personality and Social Psychology, 2004
- Mental Accounting and Consumer ChoiceMarketing Science, 1985