Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?
Top Cited Papers
- 1 June 2018
- conference paper
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 6546-6555
- https://doi.org/10.1109/cvpr.2018.00685
Abstract
The purpose of this study is to determine whether current video datasets have sufficient data for training very deep convolutional neural networks (CNNs) with spatio-temporal three-dimensional (3D) kernels. Recently, the performance levels of 3D CNNs in the field of action recognition have improved significantly. However, to date, conventional research has only explored relatively shallow 3D architectures. We examine the architectures of various 3D CNNs from relatively shallow to very deep ones on current video datasets. Based on the results of those experiments, the following conclusions could be obtained: (i) ResNet-18 training resulted in significant overfitting for UCF-101, HMDB-51, and ActivityNet but not for Kinetics. (ii) The Kinetics dataset has sufficient data for training of deep 3D CNNs, and enables training of up to 152 ResNets layers, interestingly similar to 2D ResNets on ImageNet. ResNeXt-101 achieved 78.4% average accuracy on the Kinetics test set. (iii) Kinetics pretrained simple 3D architectures outperforms complex 2D architectures, and the pretrained ResNeXt-101 achieved 94.5% and 70.2% on UCF-101 and HMDB-51, respectively. The use of 2D CNNs trained on ImageNet has produced significant progress in various tasks in image. We believe that using deep 3D CNNs together with Kinetics will retrace the successful history of 2D CNNs and ImageNet, and stimulate advances in computer vision for videos. The codes and pretrained models used in this study are publicly available 1 .Keywords
This publication has 17 references indexed in Scilit:
- Aggregated Residual Transformations for Deep Neural NetworksPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2017
- Deep Residual Learning for Image RecognitionPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2016
- Convolutional Two-Stream Network Fusion for Video Action RecognitionPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2016
- Learning Spatiotemporal Features with 3D Convolutional NetworksPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2015
- ActivityNet: A large-scale video benchmark for human activity understandingPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2015
- Action recognition with trajectory-pooled deep-convolutional descriptorsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2015
- Going deeper with convolutionsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2015
- Large-Scale Video Classification with Convolutional Neural NetworksPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2014
- Dense Trajectories and Motion Boundary Descriptors for Action RecognitionInternational Journal of Computer Vision, 2013
- 3D Convolutional Neural Networks for Human Action RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2012