RPCA-KFE: Key Frame Extraction for Video Using Robust Principal Component Analysis
- 15 June 2015
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 24 (11), 3742-3753
- https://doi.org/10.1109/tip.2015.2445572
Abstract
Key frame extraction algorithms consider the problem of selecting a subset of the most informative frames from a video to summarize its content. Several applications, such as video summarization, search, indexing, and prints from video, can benefit from extracted key frames of the video under consideration. Most approaches in this class of algorithms work directly with the input video data set, without considering the underlying low-rank structure of the data set. Other algorithms exploit the low-rank component only, ignoring the other key information in the video. In this paper, a novel key frame extraction framework based on robust principal component analysis (RPCA) is proposed. Furthermore, we target the challenging application of extracting key frames from unstructured consumer videos. The proposed framework is motivated by the observation that the RPCA decomposes an input data into: 1) a low-rank component that reveals the systematic information across the elements of the data set and 2) a set of sparse components each of which containing distinct information about each element in the same data set. The two information types are combined into a single $\ell _{1}$ -norm-based non-convex optimization problem to extract the desired number of key frames. Moreover, we develop a novel iterative algorithm to solve this optimization problem. The proposed RPCA-based framework does not require shot(s) detection, segmentation, or semantic understanding of the underlying video. Finally, experiments are performed on a variety of consumer and other types of videos. A comparison of the results obtained by our method with the ground truth and with related state-of-the-art algorithms clearly illustrates the viability of the proposed RPCA-based framework.
Keywords
Other Versions
Funding Information
- National Science Foundation (1117709, 1331852)
- Vietnam Education Foundation Fellowship
- Google Inc., Mountain View, CA, USA
This publication has 38 references indexed in Scilit:
- Heterogeneity Image Patch Index and Its Application to Consumer Video SummarizationIEEE Transactions on Image Processing, 2014
- Key frame extraction from consumer videos using epitomePublished by Institute of Electrical and Electronics Engineers (IEEE) ,2012
- VISON: VIdeo Summarization for ONline applicationsPattern Recognition Letters, 2012
- Action recognition in videos acquired by a moving camera using motion decomposition of Lagrangian particle trajectoriesPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2011
- STIMO: STIll and MOving video storyboard for the web scenarioMultimedia Tools and Applications, 2009
- Techniques for movie content analysis and skimming: tutorial and overview on video abstraction techniquesIEEE Signal Processing Magazine, 2006
- Video shot detection and condensed representation. a reviewIEEE Signal Processing Magazine, 2006
- Keyframe-based video summarization using Delaunay clusteringInternational Journal on Digital Libraries, 2006
- Video keyframe production by efficient clustering of compressed chromaticity signatures (poster session)Published by Association for Computing Machinery (ACM) ,2000
- Key frame selection to represent a videoPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2000