Expressive Facial Animation Synthesis by Learning Speech Coarticulation and Expression Spaces
- 18 September 2006
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 12 (6), 1523-1534
- https://doi.org/10.1109/tvcg.2006.90
Abstract
Synthesizing expressive facial animation is a very challenging topic within the graphics community. In this paper, we present an expressive facial animation synthesis system enabled by automated learning from facial motion capture data. Accurate 3D motions of the markers on the face of a human subject are captured while he/she recites a predesigned corpus, with specific spoken and visual expressions. We present a novel motion capture mining technique that "learns" speech coarticulation models for diphones and triphones from the recorded data. A phoneme-independent expression eigenspace (PIEES) that encloses the dynamic expression signals is constructed by motion signal processing (phoneme-based time-warping and subtraction) and principal component analysis (PCA) reduction. New expressive facial animations are synthesized as follows: First, the learned coarticulation models are concatenated to synthesize neutral visual speech according to novel speech input, then a texture-synthesis-based approach is used to generate a novel dynamic expression signal from the PIEES model, and finally the synthesized expression signal is blended with the synthesized neutral visual speech to create the final expressive facial animation. Our experiments demonstrate that the system can effectively synthesize realistic expressive facial animationThis publication has 37 references indexed in Scilit:
- Reanimating Faces in Images and VideoComputer Graphics Forum, 2003
- Facial expression space learningPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2003
- Synthetic faces: Analysis and applicationsInternational Journal of Imaging Systems and Technology, 2003
- Subtleties of facial expressions in embodied agentsThe Journal of Visualization and Computer Animation, 2002
- Principal components of expressive speech animationPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2002
- Interactive multiresolution hair modeling and editingACM Transactions on Graphics, 2002
- Expression cloningPublished by Association for Computing Machinery (ACM) ,2001
- A 3D parametric tongue model for animated speechThe Journal of Visualization and Computer Animation, 2001
- Voice puppetryPublished by Association for Computing Machinery (ACM) ,1999
- Generating Facial Expressions for SpeechCognitive Science, 1996