A Review and Meta-Analysis of Multimodal Affect Detection Systems
Top Cited Papers
- 17 February 2015
- journal article
- review article
- Published by Association for Computing Machinery (ACM) in ACM Computing Surveys
- Vol. 47 (3), 1-36
- https://doi.org/10.1145/2682899
Abstract
Affect detection is an important pattern recognition problem that has inspired researchers from several areas. The field is in need of a systematic review due to the recent influx of Multimodal (MM) affect detection systems that differ in several respects and sometimes yield incompatible results. This article provides such a survey via a quantitative review and meta-analysis of 90 peer-reviewed MM systems. The review indicated that the state of the art mainly consists of person-dependent models (62.2% of systems) that fuse audio and visual (55.6%) information to detect acted (52.2%) expressions of basic emotions and simple dimensions of arousal and valence (64.5%) with feature- (38.9%) and decision-level (35.6%) fusion techniques. However, there were also person-independent systems that considered additional modalities to detect nonbasic emotions and complex dimensions using model-level fusion techniques. The meta-analysis revealed that MM systems were consistently (85% of systems) more accurate than their best unimodal counterparts, with an average improvement of 9.83% (median of 6.60%). However, improvements were three times lower when systems were trained on natural (4.59%) versus acted data (12.7%). Importantly, MM accuracy could be accurately predicted (cross-validated R2 of 0.803) from unimodal accuracies and two system-level factors. Theoretical and applied implications and recommendations are discussed.Keywords
Funding Information
- NSF Graduate Research Fellowship under 1122374
- Bill & Melinda Gates Foundation
- National Science Foundation (NSF) (ITR 0325428, HCC 0834847, DRL 1235958)
This publication has 108 references indexed in Scilit:
- LSTM-Modeling of continuous emotions in an audiovisual affect recognition frameworkImage and Vision Computing, 2013
- Towards Efficient Multi-Modal Emotion RecognitionInternational Journal of Advanced Robotic Systems, 2013
- Music-aided affective interaction between human and service robotEURASIP Journal on Audio, Speech, and Music Processing, 2012
- Emergent Ghosts of the Emotion MachineEmotion Review, 2010
- The Experience of EmotionAnnual Review of Psychology, 2007
- Toward an affect-sensitive multimodal human-computer interactionProceedings of the IEEE, 2003
- Facial and Vocal Expressions of EmotionAnnual Review of Psychology, 2003
- On the universality and cultural specificity of emotion recognition: A meta-analysis.Psychological Bulletin, 2002
- The Nature of EmotionsAmerican Scientist, 2001
- Coherence between expressive and experiential systems in emotionCognition and Emotion, 1994