Translating Video Content to Natural Language Descriptions
- 1 December 2013
- conference paper
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 15505499,p. 433-440
- https://doi.org/10.1109/iccv.2013.61
Abstract
Humans use rich natural language to describe and communicate visual perceptions. In order to provide natural language descriptions for visual content, this paper combines two important ingredients. First, we generate a rich semantic representation of the visual content including e.g. object and activity labels. To predict the semantic representation we learn a CRF to model the relationships between different components of the visual input. And second, we propose to formulate the generation of natural language as a machine translation problem using the semantic representation as source language and the generated sentences as target language. For this we exploit the power of a parallel corpus of videos and textual descriptions and adapt statistical machine translation to translate between our two languages. We evaluate our video descriptions on the TACoS dataset, which contains video snippets aligned with sentence descriptions. Using automatic evaluation and human judgments we show significant improvements over several baseline approaches, motivated by prior work. Our translation approach also shows improvements over related work on an image description task.This publication has 12 references indexed in Scilit:
- Video Event Understanding Using Natural Language DescriptionsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2013
- Dense Trajectories and Motion Boundary Descriptors for Action RecognitionInternational Journal of Computer Vision, 2013
- Automated Textual Descriptions for a Wide Range of Video Events with 48 Human ActionsLecture Notes in Computer Science, 2012
- Towards textually describing complex video contents with audio-visual concept classifiersPublished by Association for Computing Machinery (ACM) ,2011
- Human Focused Video DescriptionPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2011
- Baby talk: Understanding and generating simple image descriptionsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2011
- Understanding videos, constructing plots learning a visually grounded storyline model from annotated videosPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2009
- Statistical machine translationACM Computing Surveys, 2008
- MosesPublished by Association for Computational Linguistics (ACL) ,2007
- A Systematic Comparison of Various Statistical Alignment ModelsComputational Linguistics, 2003