Gaze-Aware Graph Convolutional Network for Social Relation Recognition
Open Access
- 12 July 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 9, 99398-99408
- https://doi.org/10.1109/access.2021.3096553
Abstract
Social relation, as the basic relation in our daily life, is vital for social action analysis. However, how to learn the social feature between people is still not tackled. In this work, we propose a gaze-aware graph convolutional network (GA-GCN) for social relation recognition, which targets discovering the context-aware social relation inference with gaze-aware attention. To predict the gaze direction, we apply a convolutional network trained with gaze direction loss. Then, we build a graph convolutional inference module, which is a two-stream graph inference with both gaze-aware attention and distance-aware attention. The attention can pick up relevant context objects for context-aware representation. We further introduce additional scene features and construct a multiple feature fusion module, which can adaptively learn social relation representation from both scene feature and context-aware feature. Extensive experiments on the PISC and the PIPA datasets demonstrate that our GA-GCN can find interesting contextual objects and achieves state-of-the-art performances.Funding Information
- National Key Research and Development Program of China (2017YFB1002203)
- Fundamental Research Funds for the Central Universities of China (PA2020GDSK0059)
- National Nature Science Foundation of China (61503111, 61876058)
- Anhui Natural Science Foundation (1808085MF168)
This publication has 32 references indexed in Scilit:
- MPIIGaze: Real-World Dataset and Deep Appearance-Based Gaze EstimationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- Following Gaze in VideoPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2017
- Dual-Glance Model for Deciphering Social RelationshipsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2017
- Scene Graph Generation from Objects, Phrases and Region CaptionsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2017
- Joint Estimation of Human Pose and Conversational Groups from Social ScenesInternational Journal of Computer Vision, 2017
- A Domain Based Approach to Social Relation RecognitionPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2017
- Scene Graph Generation by Iterative Message PassingPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2017
- Visual Relationship Detection with Language PriorsPublished by Springer Science and Business Media LLC ,2016
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Beyond F-Formations: Determining Social Involvement in Free Standing Conversing Groups from Static ImagesPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2016