Do procedures for verbal reporting of thinking have to be reactive? A meta-analysis and recommendations for best reporting methods.
- 1 March 2011
- journal article
- research article
- Published by American Psychological Association (APA) in Psychological Bulletin
- Vol. 137 (2), 316-344
- https://doi.org/10.1037/a0021663
Abstract
Since its establishment, psychology has struggled to find valid methods for studying thoughts and subjective experiences. Thirty years ago, Ericsson and Simon (1980) proposed that participants can give concurrent verbal expression to their thoughts (think aloud) while completing tasks without changing objectively measurable performance (accuracy). In contrast, directed requests for concurrent verbal reports, such as explanations or directions to describe particular kinds of information, were predicted to change thought processes as a consequence of the need to generate this information, thus altering performance. By comparing performance of concurrent verbal reporting conditions with their matching silent control condition, Ericsson and Simon found several studies demonstrating that directed verbalization was associated with changes in performance. In contrast, the lack of effects of thinking aloud was merely suggested by a handful of experimental studies. In this article, Ericsson and Simon's model is tested by a meta-analysis of 94 studies comparing performance while giving concurrent verbalizations to a matching condition without verbalization. Findings based on nearly 3,500 participants show that the "think-aloud" effect size is indistinguishable from zero (r = -.03) and that this procedure remains nonreactive even after statistically controlling additional factors such as task type (primarily visual or nonvisual). In contrast, procedures that entail describing or explaining thoughts and actions are significantly reactive, leading to higher performance than silent control conditions. All verbal reporting procedures tend to increase times to complete tasks. These results suggest that think-aloud should be distinguished from other methods in future studies. Theoretical and practical implications are discussed.Keywords
Funding Information
- Florida State University Foundation
This publication has 67 references indexed in Scilit:
- The Effects of Aging and Dual Task Demands on Language ProductionAging, Neuropsychology, and Cognition, 2009
- Reversing the speed–IQ correlation: Intra-individual variability and attentional control in the inspection time paradigmIntelligence, 2009
- Sports forecasting: a comparison of the forecast accuracy of prediction markets, betting odds and tipstersJournal of Forecasting, 2008
- Inflexibility of experts—Reality or myth? Quantifying the Einstellung effect in chess mastersCognitive Psychology, 2008
- LEARNERS' USES OF TWO TYPES OF WRITTEN FEEDBACK ON A L2 WRITING REVISION TASKStudies in Second Language Acquisition, 2007
- THE ROLE OF LEXICAL TEMPORAL INDICATORS AND TEXT INTERACTION FORMAT IN THE INCIDENTAL ACQUISITION OF THE SPANISH FUTURE TENSEStudies in Second Language Acquisition, 2007
- When shoes become hammers: Goal-derived categorization training enhances problem-solving performance.Journal of Experimental Psychology: Learning, Memory, and Cognition, 2006
- Binary choice under instructions to select versus rejectOrganizational Behavior and Human Decision Processes, 2004
- Measuring inconsistency in meta-analysesBMJ, 2003
- How meta-analysis increases statistical power.Psychological Methods, 2003