Free-viewpoint Indoor Neural Relighting from Multi-view Stereo
- 24 September 2021
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 40 (5), 1-18
- https://doi.org/10.1145/3469842
Abstract
We introduce a neural relighting algorithm for captured indoors scenes, that allows interactive free-viewpoint navigation. Our method allows illumination to be changed synthetically, while coherently rendering cast shadows and complex glossy materials. We start with multiple images of the scene and a three-dimensional mesh obtained by multi-view stereo (MVS) reconstruction. We assume that lighting is well explained as the sum of a view-independent diffuse component and a view-dependent glossy term concentrated around the mirror reflection direction. We design a convolutional network around input feature maps that facilitate learning of an implicit representation of scene materials and illumination, enabling both relighting and free-viewpoint navigation. We generate these input maps by exploiting the best elements of both image-based and physically based rendering. We sample the input views to estimate diffuse scene irradiance, and compute the new illumination caused by user-specified light sources using path tracing. To facilitate the network's understanding of materials and synthesize plausible glossy reflections, we reproject the views and compute mirror images. We train the network on a synthetic dataset where each scene is also reconstructed with MVS. We show results of our algorithm relighting real indoor scenes and performing free-viewpoint navigation with complex and realistic glossy reflections, which so far remained out of reach for view-synthesis techniques.Keywords
Funding Information
- ERC Advanced (788065)
This publication has 66 references indexed in Scilit:
- Two-shot SVBRDF capture for stationary materialsACM Transactions on Graphics, 2015
- Depth synthesis and local warps for plausible image-based navigationACM Transactions on Graphics, 2013
- Coherent intrinsic images from photo collectionsACM Transactions on Graphics, 2012
- OptiXACM Transactions on Graphics, 2010
- Virtual Cinematography: Relighting through ComputationComputer, 2006
- Unlighting the ParthenonPublished by Association for Computing Machinery (ACM) ,2004
- Image-based reconstruction of spatial appearance and geometric detailACM Transactions on Graphics, 2003
- Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environmentsACM Transactions on Graphics, 2002
- Unstructured lumigraph renderingPublished by Association for Computing Machinery (ACM) ,2001
- Acquiring the reflectance field of a human facePublished by Association for Computing Machinery (ACM) ,2000