Visual commonsense reasoning with directional visual connections
- 28 May 2021
- journal article
- research article
- Published by Zhejiang University Press in Frontiers of Information Technology & Electronic Engineering
- Vol. 22 (5), 625-637
- https://doi.org/10.1631/fitee.2000722
Abstract
To boost research into cognition-level visual understanding, i.e., making an accurate inference based on a thorough understanding of visual details, visual commonsense reasoning (VCR) has been proposed. Compared with traditional visual question answering which requires models to select correct answers, VCR requires models to select not only the correct answers, but also the correct rationales. Recent research into human cognition has indicated that brain function or cognition can be considered as a global and dynamic integration of local neuron connectivity, which is helpful in solving specific cognition tasks. Inspired by this idea, we propose a directional connective network to achieve VCR by dynamically reorganizing the visual neuron connectivity that is contextualized using the meaning of questions and answers and leveraging the directional information to enhance the reasoning ability. Specifically, we first develop a GraphVLAD module to capture visual neuron connectivity to fully model visual content correlations. Then, a contextualization process is proposed to fuse sentence representations with visual neuron representations. Finally, based on the output of contextualized connectivity, we propose directional connectivity to infer answers and rationales, which includes a ReasonVLAD module. Experimental results on the VCR dataset and visualization analysis demonstrate the effectiveness of our method.Keywords
This publication has 29 references indexed in Scilit:
- Geometric Deep Learning on Graphs and Manifolds Using Mixture Model CNNsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2017
- NetVLAD: CNN Architecture for Weakly Supervised Place RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image SegmentationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- SSD: Single Shot MultiBox DetectorPublished by Springer Science and Business Media LLC ,2016
- Fast R-CNNPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2015
- VQA: Visual Question AnsweringPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2015
- Dynamic reorganization of brain functional networks during cognitionNeuroImage, 2015
- Structural and Functional Brain Networks: From Connections to CognitionScience, 2013
- Long Short-Term MemoryNeural Computation, 1997