Automatic 2.5-D Facial Landmarking and Emotion Annotation for Social Interaction Assistance
- 26 August 2015
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Cybernetics
- Vol. 46 (9), 2042-2055
- https://doi.org/10.1109/tcyb.2015.2461131
Abstract
People with low vision, Alzheimer's disease, and autism spectrum disorder experience difficulties in perceiving or interpreting facial expression of emotion in their social lives. Though automatic facial expression recognition (FER) methods on 2-D videos have been extensively investigated, their performance was constrained by challenges in head pose and lighting conditions. The shape information in 3-D facial data can reduce or even overcome these challenges. However, high expenses of 3-D cameras prevent their widespread use. Fortunately, 2.5-D facial data from emerging portable RGB-D cameras provide a good balance for this dilemma. In this paper, we propose an automatic emotion annotation solution on 2.5-D facial data collected from RGB-D cameras. The solution consists of a facial landmarking method and a FER method. Specifically, we propose building a deformable partial face model and fit the model to a 2.5-D face for localizing facial landmarks automatically. In FER, a novel action unit (AU) space-based FER method has been proposed. Facial features are extracted using landmarks and further represented as coordinates in the AU space, which are classified into facial expressions. Evaluated on three publicly accessible facial databases, namely EURECOM, FRGC, and Bosphorus databases, the proposed facial landmarking and expression recognition methods have achieved satisfactory results. Possible real-world applications using our algorithms have also been discussed.Keywords
Funding Information
- National Nature Science Foundation of China (61303121, 11401464)
- Microsoft Research Asia Collaborative Research Award
- French Research Agency, Agence Nationale de Recherche through the Jemime Project (ANR-13-CORD-004-02)
This publication has 65 references indexed in Scilit:
- Ultra-deep sequencing enables high-fidelity recovery of biodiversity for bulk arthropod samples without PCR amplificationGigaScience, 2013
- Realtime performance-based facial animationACM Transactions on Graphics, 2011
- The use of innovative computer technology for teaching social communication to individuals with autism spectrum disordersResearch in Autism Spectrum Disorders, 2011
- Facial expression recognition based on Local Binary Patterns: A comprehensive studyImage and Vision Computing, 2009
- Physiology-based affect recognition for computer-assisted intervention of children with Autism Spectrum DisorderInternational Journal of Human-Computer Studies, 2008
- Face recognition based on fitting a 3D morphable modelIEEE Transactions on Pattern Analysis and Machine Intelligence, 2003
- Automatic facial expression analysis: a surveyPattern Recognition, 2003
- A comparison of methods for multiclass support vector machinesIEEE Transactions on Neural Networks, 2002
- COSMOS-A representation scheme for 3D free-form objectsIEEE Transactions on Pattern Analysis and Machine Intelligence, 1997
- Active Shape Models-Their Training and ApplicationComputer Vision and Image Understanding, 1995