Accelerating Video Captioning on Heterogeneous System Architectures
Open Access
- 25 May 2022
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Architecture and Code Optimization
- Vol. 19 (3), 1-25
- https://doi.org/10.1145/3527609
Abstract
Video captioning is a core technology to many important applications, such as AI-assisted medical diagnosis, video question answering, storytelling through videos, and lip-reading. Video captioning employs a hybrid CNN + RNN model. Accelerating such a hybrid model on a heterogeneous system is challenging for two reasons. First, CNN and RNN exhibit very different computing behaviors, making the mapping between computation and heterogeneous devices difficult. Second, data dependency exists between the CNN and RNN within a video frame and between adjacent RNNs across video frames. These data dependencies prohibit the full parallelization of the hybrid model. The issues also include the utilization of accelerator resources, which is critical to maximizing the performance. In this work, we propose a fine-grained scheduling scheme for mapping computation and devices within a video frame, and a pipeline scheduling scheme for exploiting maximum parallelism between the execution of the video frames. In addition, we propose two capacity-guided scheduling methods. On the server, the concurrent kernel execution mechanism is exploited for improving GPU utilization. On the edge platform, we rearrange CNN computation among the CPU and EdgeTPUs guided by the EdgeTPU’s SRAM capacity so that balanced computation is achieved and off-chip memory overhead is minimized. Experimental results show that our scheduling scheme improves video captioning performance by up to 3.24 \( \times \) with CPU + GPU collaboration over the GPU-only execution. On an edge platform with an ARM CPU and two EdgeTPUs, our CPU + EdgeTPU scheduling exhibits outstanding performance, which achieves up to 54.9 \( \times \) speedup compared to using ARM CPU only and can perform video captioning of 59 frames per second.
Keywords
Funding Information
- Ministry of Science and Technology (MOST-110-2221-E-001-007-MY2, MOST-110-2221-E-002-072-MY2, and MOST-110-2221-E-001-004)
This publication has 52 references indexed in Scilit:
- Eyeriss: A Spatial Architecture for Energy-Efficient Dataflow for Convolutional Neural NetworksPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2016
- MinervaACM SIGARCH Computer Architecture News, 2016
- CambriconACM SIGARCH Computer Architecture News, 2016
- EyerissACM SIGARCH Computer Architecture News, 2016
- Multi-view Convolutional Neural Networks for 3D Shape RecognitionPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2015
- Sequence to Sequence -- Video to TextPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2015
- U-Net: Convolutional Networks for Biomedical Image SegmentationPublished by Springer Science and Business Media LLC ,2015
- Translating Videos to Natural Language Using Deep Recurrent Neural NetworksPublished by Association for Computational Linguistics (ACL) ,2015
- On the Properties of Neural Machine Translation: Encoder–Decoder ApproachesPublished by Association for Computational Linguistics (ACL) ,2014
- Long Short-Term MemoryNeural Computation, 1997