ExpressEar
- 9 September 2021
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
- Vol. 5 (3), 1-28
- https://doi.org/10.1145/3478085
Abstract
Continuous and unobtrusive monitoring of facial expressions holds tremendous potential to enable compelling applications in a multitude of domains ranging from healthcare and education to interactive systems. Traditional, vision-based facial expression recognition (FER) methods, however, are vulnerable to external factors like occlusion and lighting, while also raising privacy concerns coupled with the impractical requirement of positioning the camera in front of the user at all times. To bridge this gap, we propose ExpressEar, a novel FER system that repurposes commercial earables augmented with inertial sensors to capture fine-grained facial muscle movements. Following the Facial Action Coding System (FACS), which encodes every possible expression in terms of constituent facial movements called Action Units (AUs), ExpressEar identifies facial expressions at the atomic level. We conducted a user study (N=12) to evaluate the performance of our approach and found that ExpressEar can detect and distinguish between 32 Facial AUs (including 2 variants of asymmetric AUs), with an average accuracy of 89.9% for any given user. We further quantify the performance across different mobile scenarios in presence of additional face-related activities. Our results demonstrate ExpressEar's applicability in the real world and open up research opportunities to advance its practical adoption.Keywords
This publication has 59 references indexed in Scilit:
- Activity recognition using cell phone accelerometersACM SIGKDD Explorations Newsletter, 2011
- Are there ethnic differences of facial movements between Europeans and Asians?British Journal of Plastic Surgery, 2005
- Noninvasive Brain-Actuated Control of a Mobile Robot by Human EEGIEEE Transactions on Biomedical Engineering, 2004
- Automatic facial expression analysis: a surveyPattern Recognition, 2003
- Human facial expressions as adaptations: Evolutionary questions in facial expression researchAmerican Journal of Physical Anthropology, 2001
- Algorithms on Strings, Trees, and Sequences: Computer Science and Computational BiologyJournal of the American Statistical Association, 1999
- Gender Differences in Nonverbal Correlates of Social StatusPersonality and Social Psychology Bulletin, 1996
- Gender-based expectancies and observer judgments of smilingJournal of Nonverbal Behavior, 1995
- The Production and Judgment of Smile MagnitudeThe Journal of General Psychology, 1978
- Condylar Displacement Recordings as an Analog of Mandibular MovementsJournal of Dental Research, 1968