Video-based Facial Micro-Expression Analysis: A Survey of Datasets, Features and Algorithms
Top Cited Papers
- 19 March 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 44 (01628828), 1
- https://doi.org/10.1109/tpami.2021.3067464
Abstract
Unlike the conventional facial expressions, micro-expressions are involuntary and transient facial expressions capable of revealing the genuine emotions that people attempt to hide. Therefore, they can provide important information in a broad range of applications such as lie detection, criminal detection, etc. Since micro-expressions are transient and of low intensity, however, their detection and recognition is difficult and relies heavily on expert experiences. Due to its intrinsic particularity and complexity, video-based micro-expression analysis is attractive but challenging, and has recently become an active area of research. Although there have been numerous developments in this area, thus far there has been no comprehensive survey that provides researchers with a systematic overview of these developments with a unified evaluation. Accordingly, in this survey paper, we first highlight the key differences between macro- and micro-expressions, then use these differences to guide our research survey of video-based micro-expression analysis in a cascaded structure, encompassing the neuropsychological basis, datasets, features, spotting algorithms, recognition algorithms, applications and evaluation of state-of-the-art approaches. For each aspect, the basic techniques, advanced developments and major challenges are addressed and discussed. Furthermore, after considering the limitations of existing micro-expression datasets, we present and release a new dataset — called micro-and-macro expression warehouse (MMEW) — containing more video samples and more labeled emotion types. We then perform a unified comparison of representative methods on CAS(ME) $^2$ for spotting, and on MMEW and SAMM for recognition, respectively. Finally, some potential future research directions are explored and outlined.
Funding Information
- National Natural Science Foundation of China (61725204)
This publication has 114 references indexed in Scilit:
- LIBSVMACM Transactions on Intelligent Systems and Technology, 2011
- Reading Between the LiesPsychological Science, 2008
- From flawed self‐assessment to blatant whoppers: the utility of voluntary and involuntary behavior in detecting deceptionBehavioral Sciences & the Law, 2006
- Silent talker: a new computer-based system for the analysis of facial cues to deceptionApplied Cognitive Psychology, 2006
- What are emotions? And how can they be measured?Social Science Information, 2005
- Multiresolution gray-scale and rotation invariant texture classification with local binary patternsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2002
- Active appearance modelsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2001
- Differentiating emotion elicited and deliberate emotional facial expressionsEuropean Journal of Social Psychology, 1990
- The neuropsychology of facial expression: A review of the neurological and psychological mechanisms for producing facial expressions.Psychological Bulletin, 1984
- Detecting deception from the body or face.Journal of Personality and Social Psychology, 1974