Visual Translation Embedding Network for Visual Relation Detection
- 1 July 2017
- conference paper
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 10636919,p. 3107-3115
- https://doi.org/10.1109/cvpr.2017.331
Abstract
Visual relations, such as person ride bike and bike next to car, offer a comprehensive scene understanding of an image, and have already shown their great utility in connecting computer vision and natural language. However, due to the challenging combinatorial complexity of modeling subject-predicate-object relation triplets, very little work has been done to localize and predict visual relations. Inspired by the recent advances in relational representation learning of knowledge bases and convolutional object detection networks, we propose a Visual Translation Embedding network (VTransE) for visual relation detection. VTransE places objects in a low-dimensional relation space where a relation can be modeled as a simple vector translation, i.e., subject + predicate ≈ object. We propose a novel feature extraction layer that enables object-relation knowledge transfer in a fully-convolutional fashion that supports training and inference in a single forward/backward pass. To the best of our knowledge, VTransE is the first end-toend relation detection network. We demonstrate the effectiveness of VTransE over other state-of-the-art methods on two large-scale datasets: Visual Relationship and Visual Genome. Note that even though VTransE is a purely visual model, it is still competitive to the Lu's multi-modal model with language priors [27].Keywords
This publication has 27 references indexed in Scilit:
- Deep visual-semantic alignments for generating image descriptionsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2015
- Learning semantic relationships for better action retrieval in imagesPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2015
- VisKE: Visual knowledge extraction and question answering by visual verification of relation phrasesPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2015
- Question Answering over Freebase with Multi-Column Convolutional Neural NetworksPublished by Association for Computational Linguistics (ACL) ,2015
- Rich Feature Hierarchies for Accurate Object Detection and Semantic SegmentationPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2014
- Discriminative Models for Multi-Class Object LayoutInternational Journal of Computer Vision, 2011
- Every Picture Tells a Story: Generating Sentences from ImagesLecture Notes in Computer Science, 2010
- Efficient Object Category Recognition Using ClassemesLecture Notes in Computer Science, 2010
- Beyond Nouns: Exploiting Prepositions and Comparative Adjectives for Learning Visual ClassifiersLecture Notes in Computer Science, 2008
- Representation of Manipulable Man-Made Objects in the Dorsal StreamNeuroImage, 2000