GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction
- 1 June 2019
- conference paper
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1155-1164
- https://doi.org/10.1109/cvpr.2019.00125
Abstract
In the past few years, a lot of work has been done towards reconstructing the 3D facial structure from single images by capitalizing on the power of Deep Convolutional Neural Networks (DCNNs). In the most recent works, differentiable renderers were employed in order to learn the relationship between the facial identity features and the parameters of a 3D morphable model for shape and texture. The texture features either correspond to components of a linear texture space or are learned by auto-encoders directly from in-the-wild images. In all cases, the quality of the facial texture reconstruction of the state-of-the-art methods is still not capable of modeling textures in high fidelity. In this paper, we take a radically different approach and harness the power of Generative Adversarial Networks (GANs) and DCNNs in order to reconstruct the facial texture and shape from single images. That is, we utilize GANs to train a very powerful generator of facial texture in UV space. Then, we revisit the original 3D Morphable Models (3DMMs) fitting approaches making use of non-linear optimization to find the optimal latent parameters that best reconstruct the test image but under a new perspective. We optimize the parameters with the supervision of pretrained deep identity features through our end-to-end differentiable framework. We demonstrate excellent results in photorealistic and identity preserving 3D face reconstructions and achieve for the first time, to the best of our knowledge, facial texture reconstruction with high-frequency details.Keywords
This publication has 30 references indexed in Scilit:
- 3D Face Reconstruction by Learning from Synthetic DataPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2016
- Demo of Face2FacePublished by Association for Computing Machinery (ACM) ,2016
- Face Alignment Across Large Poses: A 3D SolutionPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2016
- FaceNet: A unified embedding for face recognition and clusteringPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2015
- The florence 2D/3D hybrid face datasetPublished by Association for Computing Machinery (ACM) ,2011
- Principal Component AnalysisPublished by Springer Science and Business Media LLC ,2011
- Face recognition based on fitting a 3D morphable modelIEEE Transactions on Pattern Analysis and Machine Intelligence, 2003
- Face Identification by Fitting a 3D Morphable Model Using Linear Shape and Texture Error FunctionsLecture Notes in Computer Science, 2002
- A morphable model for the synthesis of 3D facesPublished by Association for Computing Machinery (ACM) ,1999
- Method for registration of 3-D shapesPublished by SPIE-Intl Soc Optical Eng ,1992