The Role of Iconic Gestures in Speech Disambiguation: ERP Evidence
- 1 July 2007
- journal article
- Published by MIT Press in Journal of Cognitive Neuroscience
- Vol. 19 (7), 1175-1192
- https://doi.org/10.1162/jocn.2007.19.7.1175
Abstract
The present series of experiments explored the extent to which iconic gestures convey information not found in speech. Electroencephalogram (EEG) was recorded as participants watched videos of a person gesturing and speaking simultaneously. The experimental sentences contained an unbalanced homonym in the initial part of the sentence (e.g., She controlled the ball …) and were disambiguated at a target word in the subsequent clause (which during the game … vs. which during the dance …). Coincident with the initial part of the sentence, the speaker produced an iconic gesture which supported either the dominant or the subordinate meaning. Event-related potentials were time-locked to the onset of the target word. In Experiment 1, participants were explicitly asked to judge the congruency between the initial homonym-gesture combination and the subsequent target word. The N400 at target words was found to be smaller after a congruent gesture and larger after an incongruent gesture, suggesting that listeners can use gestural information to disambiguate speech. Experiment 2 replicated the results using a less explicit task, indicating that the disambiguating effect of gesture is somewhat task-independent. Unrelated grooming movements were added to the paradigm in Experiment 3. The N400 at subordinate targets was found to be smaller after subordinate gestures and larger after dominant gestures as well as grooming, indicating that an iconic gesture can facilitate the processing of a lesser frequent word meaning. The N400 at dominant targets no longer varied as a function of the preceding gesture in Experiment 3, suggesting that the addition of meaningless movements weakened the impact of gesture. Thus, the integration of gesture and speech in comprehension does not appear to be an obligatory process but is modulated by situational factors such as the amount of observed meaningful hand movements.Keywords
This publication has 37 references indexed in Scilit:
- Communicating hands: ERPs elicited by meaningful symbolic hand posturesNeuroscience Letters, 2004
- Cues to deception.Psychological Bulletin, 2003
- An experimental investigation of the role of different types of iconic gesture in communicationRecurrent Gestures, 2002
- An experimental investigation of some properties of individual iconic gestures that mediate their communicative powerBritish Journal of Psychology, 2002
- Mapping the Range of Information Contained in the Iconic Hand Gestures that Accompany Spontaneous SpeechJournal of Language and Social Psychology, 1999
- Gestures convey substantive information about a child’s thoughts to ordinary listenersDevelopmental Science, 1999
- Do iconic hand gestures really contribute anything to the semantic information conveyed by speech? An experimental investigationSemiotica, 1999
- Assessing knowledge conveyed in gesture: Do teachers have the upper hand?Journal of Educational Psychology, 1997
- The N400 as a function of the level of processingPsychophysiology, 1995
- On methods in the analysis of profile dataPsychometrika, 1959