Deep incremental learning for efficient high-fidelity face tracking
- 4 December 2018
- journal article
- conference paper
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 37 (6), 1-12
- https://doi.org/10.1145/3272127.3275101
Abstract
In this paper, we present an incremental learning framework for efficient and accurate facial performance tracking. Our approach is to alternate the modeling step, which takes tracked meshes and texture maps to train our deep learning-based statistical model, and the tracking step, which takes predictions of geometry and texture our model infers from measured images and optimize the predicted geometry by minimizing image, geometry and facial landmark errors. Our Geo-Tex VAE model extends the convolutional variational autoencoder for face tracking, and jointly learns and represents deformations and variations in geometry and texture from tracked meshes and texture maps. To accurately model variations in facial geometry and texture, we introduce the decomposition layer in the Geo-Tex VAE architecture which decomposes the facial deformation into global and local components. We train the global deformation with a fully-connected network and the local deformations with convolutional layers. Despite running this model on each frame independently - thereby enabling a high amount of parallelization - we validate that our framework achieves sub-millimeter accuracy on synthetic data and outperforms existing methods. We also qualitatively demonstrate high-fidelity, long-duration facial performance tracking on several actors.Keywords
This publication has 57 references indexed in Scilit:
- Lightweight binocular facial performance capture under uncontrolled lightingACM Transactions on Graphics, 2012
- Realtime performance-based facial animationACM Transactions on Graphics, 2011
- Interactive region-based linear 3D face modelsACM Transactions on Graphics, 2011
- High-quality passive facial performance capture using anchor framesACM Transactions on Graphics, 2011
- Leveraging motion capture and 3D scanning for high-fidelity facial performance acquisitionACM Transactions on Graphics, 2011
- High resolution passive facial performance captureACM Transactions on Graphics, 2010
- Example-based facial riggingACM Transactions on Graphics, 2010
- Face transfer with multilinear modelsACM Transactions on Graphics, 2005
- Distinctive Image Features from Scale-Invariant KeypointsInternational Journal of Computer Vision, 2004
- Poisson image editingACM Transactions on Graphics, 2003