Multi-Objective Based Spatio-Temporal Feature Representation Learning Robust to Expression Intensity Variations for Facial Expression Recognition
- 19 April 2017
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Affective Computing
- Vol. 10 (2), 223-236
- https://doi.org/10.1109/taffc.2017.2695999
Abstract
Facial expression recognition (FER) is increasingly gaining importance in various emerging affective computing applications. In practice, achieving accurate FER is challenging due to the large amount of inter-personal variations such as expression intensity variations. In this paper, we propose a new spatio-temporal feature representation learning for FER that is robust to expression intensity variations. The proposed method utilizes representative expression-states (e.g., onset, apex and offset of expressions) which can be specified in facial sequences regardless of the expression intensity. The characteristics of facial expressions are encoded in two parts in this paper. As the first part, spatial image characteristics of the representative expression-state frames are learned via a convolutional neural network. Five objective terms are proposed to improve the expression class separability of the spatial feature representation. In the second part, temporal characteristics of the spatial feature representation in the first part are learned with a long short-term memory of the facial expression. Comprehensive experiments have been conducted on a deliberate expression dataset (MMI) and a spontaneous micro-expression dataset (CASME II). Experimental results showed that the proposed method achieved higher recognition rates in both datasets compared to the state-of-the-art methods.Keywords
Funding Information
- National Research Foundation of Korea
- Korea government (2015R1A2A2A01005724)
This publication has 42 references indexed in Scilit:
- A Dynamic Appearance Descriptor Approach to Facial Actions Temporal ModelingIEEE Transactions on Cybernetics, 2013
- Robust Facial Expression Recognition via Compressive SensingSensors, 2012
- Gabor wavelets and General Discriminant Analysis for face identification and verificationImage and Vision Computing, 2007
- Facial Expression AnalysisPublished by Springer Science and Business Media LLC ,2005
- Fast normalized cross correlation for defect detectionPattern Recognition Letters, 2003
- Automatic facial expression analysis: a surveyPattern Recognition, 2003
- Face recognition by independent component analysisIEEE Transactions on Neural Networks, 2002
- Classifying facial actionsIEEE Transactions on Pattern Analysis and Machine Intelligence, 1999
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- Long Short-Term MemoryNeural Computation, 1997