Audio-Visual Speech Cue Combination
Open Access
- 16 April 2010
- journal article
- research article
- Published by Public Library of Science (PLoS) in PLOS ONE
- Vol. 5 (4), e10217
- https://doi.org/10.1371/journal.pone.0010217
Abstract
Different sources of sensory information can interact, often shaping what we think we have seen or heard. This can enhance the precision of perceptual decisions relative to those made on the basis of a single source of information. From a computational perspective, there are multiple reasons why this might happen, and each predicts a different degree of enhanced precision. Relatively slight improvements can arise when perceptual decisions are made on the basis of multiple independent sensory estimates, as opposed to just one. These improvements can arise as a consequence of probability summation. Greater improvements can occur if two initially independent estimates are summated to form a single integrated code, especially if the summation is weighted in accordance with the variance associated with each independent estimate. This form of combination is often described as a Bayesian maximum likelihood estimate. Still greater improvements are possible if the two sources of information are encoded via a common physiological process. Here we show that the provision of simultaneous audio and visual speech cues can result in substantial sensitivity improvements, relative to single sensory modality based decisions. The magnitude of the improvements is greater than can be predicted on the basis of either a Bayesian maximum likelihood estimate or a probability summation. Our data suggest that primary estimates of speech content are determined by a physiological process that takes input from both visual and auditory processing, resulting in greater sensitivity than would be possible if initially independent audio and visual estimates were formed and then subsequently combined.Keywords
This publication has 20 references indexed in Scilit:
- Meaningful auditory information enhances perception of visual biological motionJournal of Vision, 2009
- Lip-Reading Aids Word Recognition Most in Moderate Noise: A Bayesian Explanation Using High-Dimensional Feature SpacePLOS ONE, 2009
- Visual search for a target changing in synchrony with an auditory signalProceedings Of The Royal Society B-Biological Sciences, 2005
- Assessing automaticity in audiovisual speech integration: evidence from the speeded classification taskCognition, 2004
- The integration of auditory and visual motion signals at thresholdPerception & Psychophysics, 2003
- Combining Sensory Information: Mandatory Fusion Within, but Not Between, SensesScience, 2002
- Humans integrate visual and haptic information in a statistically optimal fashionNature, 2002
- Sound alters visual motion perceptionNature, 1997
- Two stages of visual processing for radial and circular motionNature, 1995
- Hearing lips and seeing voicesNature, 1976