Multiframe-to-Multiframe Network for Video Denoising
- 3 May 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 24 (15209210), 2164-2178
- https://doi.org/10.1109/tmm.2021.3077140
Abstract
Most existing studies performed video denoising by using multiple adjacent noisy frames to recover one clean frame; however, despite achieving relatively good quality for each individual frame, these approaches may result in visual flickering when the denoised frames are considered in sequence. In this paper, instead of separately restoring each clean frame, we propose a multiframe-to-multiframe (MM) denoising scheme that simultaneously recovers multiple clean frames from consecutive noisy frames. The proposed MM denoising scheme uses a training strategy that optimizes the denoised video from both the spatial and temporal dimensions, enabling better temporal consistency in the denoised video. Furthermore, we present an MM network (MMNet), which adopts a spatiotemporal convolutional architecture that considers both the interframe similarity and single-frame characteristics. Benefiting from the underlying parallel mechanism of the MM denoising scheme, MMNet achieves a highly competitive denoising efficiency. Extensive analyses and experiments demonstrate that MMNet outperforms the state-of-the-art video denoising methods, yielding temporal consistency improvements of at least 13.3% and running more than 2 times faster than the other methods.Keywords
Funding Information
- National Natural Science Foundation of China (61727809)
- Special Fund for Key Program of Science, and Technology of Anhui Province (201903a05020022, 201903c08020002)
This publication has 63 references indexed in Scilit:
- Trainable Nonlinear Reaction Diffusion: A Flexible Framework for Fast and Effective Image RestorationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Deep Residual Learning for Image RecognitionPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2016
- Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural NetworkPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2016
- Patch-Based Video Denoising With Optical Flow EstimationIEEE Transactions on Image Processing, 2016
- Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet ClassificationPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2015
- Nonlocal Transform-Domain Filter for Volumetric Data Denoising and ReconstructionIEEE Transactions on Image Processing, 2012
- Video Denoising, Deblocking, and Enhancement Through Separable 4-D Nonlocal Spatiotemporal TransformsIEEE Transactions on Image Processing, 2012
- 3D Convolutional Neural Networks for Human Action RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2012
- Image Denoising by Sparse 3-D Transform-Domain Collaborative FilteringIEEE Transactions on Image Processing, 2007
- Fields of Experts: A Framework for Learning Image PriorsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2005