Deep3D reconstruction: methods, data, and challenges
- 28 May 2021
- journal article
- review article
- Published by Zhejiang University Press in Frontiers of Information Technology & Electronic Engineering
- Vol. 22 (5), 652-672
- https://doi.org/10.1631/fitee.2000068
Abstract
Three-dimensional (3D) reconstruction of shapes is an important research topic in the fields of computer vision, computer graphics, pattern recognition, and virtual reality. Existing 3D reconstruction methods usually suffer from two bottlenecks: (1) they involve multiple manually designed states which can lead to cumulative errors, but can hardly learn semantic features of 3D shapes automatically; (2) they depend heavily on the content and quality of images, as well as precisely calibrated cameras. As a result, it is difficult to improve the reconstruction accuracy of those methods. 3D reconstruction methods based on deep learning overcome both of these bottlenecks by automatically learning semantic features of 3D shapes from low-quality images using deep networks. However, while these methods have various architectures, in-depth analysis and comparisons of them are unavailable so far. We present a comprehensive survey of 3D reconstruction methods based on deep learning. First, based on different deep learning model architectures, we divide 3D reconstruction methods based on deep learning into four types, recurrent neural network, deep autoencoder, generative adversarial network, and convolutional neural network based methods, and analyze the corresponding methodologies carefully. Second, we investigate four representative databases that are commonly used by the above methods in detail. Third, we give a comprehensive comparison of 3D reconstruction methods based on deep learning, which consists of the results of different methods with respect to the same database, the results of each method with respect to different databases, and the robustness of each method with respect to the number of views. Finally, we discuss future development of 3D reconstruction methods based on deep learning.Keywords
This publication has 63 references indexed in Scilit:
- Learning 3D Object Templates by Quantizing Geometry and Appearance SpacesIEEE Transactions on Pattern Analysis and Machine Intelligence, 2014
- The Pascal Visual Object Classes Challenge: A RetrospectiveInternational Journal of Computer Vision, 2014
- A search-classify approach for cluttered indoor scene understandingACM Transactions on Graphics, 2012
- An interactive approach to semantic modeling of indoor scenes with an RGBD cameraACM Transactions on Graphics, 2012
- Indoor Segmentation and Support Inference from RGBD ImagesLecture Notes in Computer Science, 2012
- SSD: Smooth Signed Distance Surface ReconstructionComputer Graphics Forum, 2011
- Carved Visual Hulls for Image-Based ModelingInternational Journal of Computer Vision, 2008
- Carved Visual Hulls for Image-Based ModelingLecture Notes in Computer Science, 2006
- Algorithm 778: L-BFGS-BACM Transactions on Mathematical Software, 1997
- Long Short-Term MemoryNeural Computation, 1997