Painting-to-3D model alignment via discriminative visual elements
- 1 March 2014
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 33 (2), 1-14
- https://doi.org/10.1145/2591009
Abstract
This article describes a technique that can reliably align arbitrary 2D depictions of an architectural site, including drawings, paintings, and historical photographs, with a 3D model of the site. This is a tremendously difficult task, as the appearance and scene structure in the 2D depictions can be very different from the appearance and geometry of the 3D model, for example, due to the specific rendering style, drawing error, age, lighting, or change of seasons. In addition, we face a hard search problem: the number of possible alignments of the painting to a large 3D model, such as a partial reconstruction of a city, is huge. To address these issues, we develop a new compact representation of complex 3D scenes. The 3D model of the scene is represented by a small set of discriminative visual elements that are automatically learned from rendered views. Similar to object detection, the set of visual elements, as well as the weights of individual features for each element, are learned in a discriminative fashion. We show that the learned visual elements are reliably matched in 2D depictions of the scene despite large variations in rendering style (e.g., watercolor, sketch, historical photograph) and structural changes (e.g., missing scene parts, large occluders) of the scene. We demonstrate an application of the proposed approach to automatic rephotography to find an approximate viewpoint of historical paintings and photographs with respect to a 3D model of the site. The proposed alignment procedure is validated via a human user study on a new database of paintings and sketches spanning several sites. The results demonstrate that our algorithm produces significantly better alignments than several baseline methods.Keywords
Funding Information
- European Institute of Innovation and Technology
- MSR-INRIA laboratory
- Agence Nationale de la Recherche
This publication has 44 references indexed in Scilit:
- A Survey of Urban ReconstructionComputer Graphics Forum, 2013
- Pegasos: primal estimated sub-gradient solver for SVMMathematical Programming, 2010
- Unstructured video-based renderingACM Transactions on Graphics, 2010
- Computational rephotographyACM Transactions on Graphics, 2010
- Automated recognition of 3D CAD model objects in laser scans and calculation of as-built dimensions for dimensional compliance control in constructionAdvanced Engineering Informatics, 2010
- A geometrical analysis of multiple viewpoint perspective in the work of Giovanni Battista Piranesi: an application of geometric restitution of perspectiveThe Journal of Architecture, 2008
- Deep photoACM Transactions on Graphics, 2008
- Image Alignment and Stitching: A TutorialFoundations and Trends® in Computer Graphics and Vision, 2007
- Photo tourismACM Transactions on Graphics, 2006
- The Divergence and Bhattacharyya Distance Measures in Signal SelectionIEEE Transactions on Communications, 1967