Image Generation from Scene Graphs
Top Cited Papers
- 1 June 2018
- conference paper
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1219-1228
- https://doi.org/10.1109/cvpr.2018.00133
Abstract
To truly understand the visual world our models should be able not only to recognize images but also generate them. To this end, there has been exciting recent progress on generating images from natural language descriptions. These methods give stunning results on limited domains such as descriptions of birds or flowers, but struggle to faithfully reproduce complex sentences with many objects and relationships. To overcome this limitation we propose a method for generating images from scene graphs, enabling explicitly reasoning about objects and their relationships. Our model uses graph convolution to process input graphs, computes a scene layout by predicting bounding boxes and segmentation masks for objects, and converts the layout to an image with a cascaded refinement network. The network is trained adversarially against a pair of discriminators to ensure realistic outputs. We validate our approach on Visual Genome and COCO-Stuff, where qualitative results, ablations, and user studies demonstrate our method's ability to generate complex images with multiple objects.Keywords
This publication has 25 references indexed in Scilit:
- Image retrieval using scene graphsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2015
- Going deeper with convolutionsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Generating Semantically Precise Scene Graphs from Textual Descriptions for Improved Image RetrievalPublished by Association for Computational Linguistics (ACL) ,2015
- Text to 3D Scene Generation with Rich Lexical GroundingPublished by Association for Computational Linguistics (ACL) ,2015
- DeepWalkPublished by Association for Computing Machinery (ACM) ,2014
- Learning Spatial Knowledge for Text to 3D Scene GenerationPublished by Association for Computational Linguistics (ACL) ,2014
- A new model for learning in graph domainsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2006
- Learning task-dependent distributed representations by backpropagation through structurePublished by Institute of Electrical and Electronics Engineers (IEEE) ,2002
- Supervised neural networks for the classification of structuresIEEE Transactions on Neural Networks, 1997