Video Denoising via Dynamic Video Layering

Video denoising refers to the problem of removing “noise” from a video sequence. Here, the term “noise” is used in a broad sense to refer to any corruption or outlier or interference that is not the quantity of interest. We develop a novel solution framework, which we call layering denoising (LD), for denoising highly noisy or otherwise corrupted videos that are well modeled as the sum of a low-rank matrix plus a sparse matrix. We show that the performance of existing state-of-the-art denoisers can be significantly improved (especially in large noise settings) if the video is first decomposed into the two layers and the denoiser is applied on each layer separately. Our proposed solution uses a recursive projected compressive sensing (ReProCS) based algorithm for the layering task and video block matching and three-dimensional filtering for denoising each layer. We show the power of our proposed approach, ReProCS-LD, using exhaustive experimental comparisons.

[1]  Shuicheng Yan,et al.  Online Robust PCA via Stochastic Optimization , 2013, NIPS.

[2]  Laura Balzano,et al.  Incremental gradient on the Grassmannian for online foreground and background separation in subsampled video , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[3]  Enhong Chen,et al.  Image Denoising and Inpainting with Deep Neural Networks , 2012, NIPS.

[4]  Michael Elad,et al.  Image Denoising Via Sparse and Redundant Representations Over Learned Dictionaries , 2006, IEEE Transactions on Image Processing.

[5]  Saeed Gazor,et al.  Video denoising in three-dimensional complex wavelet domain using a doubly stochastic modelling , 2012 .

[6]  Jean-Michel Morel,et al.  Image denoising by non-local averaging , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[7]  Yoram Bresler,et al.  Online Sparsifying Transform Learning— Part I: Algorithms , 2015, IEEE Journal of Selected Topics in Signal Processing.

[8]  Namrata Vaswani,et al.  Recursive Robust PCA or Recursive Sparse Recovery in Large but Structured Noise , 2012, IEEE Transactions on Information Theory.

[9]  Xiaokang Yang,et al.  Deep RNNs for video denoising , 2016, Optical Engineering + Applications.

[10]  Yoram Bresler,et al.  Online Sparsifying Transform Learning—Part II: Convergence Analysis , 2015, IEEE Journal of Selected Topics in Signal Processing.

[11]  Praneeth Narayanamurthy,et al.  Nearly Optimal Robust Subspace Tracking and Dynamic Robust PCA , 2017, ICML 2018.

[12]  Guillermo Sapiro,et al.  Non-local sparse models for image restoration , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[13]  Karen O. Egiazarian,et al.  Pointwise Shape-Adaptive DCT for High-Quality Denoising and Deblocking of Grayscale and Color Images , 2007, IEEE Transactions on Image Processing.

[14]  Prateek Jain,et al.  Non-convex Robust PCA , 2014, NIPS.

[15]  Thierry Bouwmans,et al.  Robust PCA via Principal Component Pursuit: A review for a comparative evaluation in video surveillance , 2014, Comput. Vis. Image Underst..

[16]  Xiaodong Li,et al.  Stable Principal Component Pursuit , 2010, 2010 IEEE International Symposium on Information Theory.

[17]  Peyman Milanfar,et al.  Clustering-Based Denoising With Locally Learned Dictionaries , 2009, IEEE Transactions on Image Processing.

[18]  Namrata Vaswani,et al.  Real-time Robust Principal Components' Pursuit , 2010, 2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[19]  Zuowei Shen,et al.  Robust Video Restoration by Joint Sparse and Low Rank Matrix Approximation , 2011, SIAM J. Imaging Sci..

[20]  Yi Ma,et al.  Robust principal component analysis? , 2009, JACM.

[21]  Stefan Harmeling,et al.  Image denoising: Can plain neural networks compete with BM3D? , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  Michal Joachimiak,et al.  Multiview 3D video denoising in sliding 3D DCT domain , 2012, 2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO).

[23]  Honglak Lee,et al.  Adaptive Multi-Column Deep Neural Networks with Application to Robust Image Denoising , 2013, NIPS.

[24]  Michael J. Black,et al.  A Framework for Robust Subspace Learning , 2003, International Journal of Computer Vision.

[25]  Yanjun Li,et al.  Joint Adaptive Sparsity and Low-Rankness on the Fly: An Online Tensor Reconstruction Scheme for Video Denoising , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[26]  Morteza Mardani,et al.  Dynamic Anomalography: Tracking Network Anomalies Via Sparsity and Low Rank , 2012, IEEE Journal of Selected Topics in Signal Processing.

[27]  Peyman Milanfar,et al.  Is Denoising Dead? , 2010, IEEE Transactions on Image Processing.

[28]  Motaz El-Saban,et al.  FRPCA: Fast Robust Principal Component Analysis for online observations , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[29]  William T. Freeman,et al.  A High-Quality Video Denoising Algorithm Based on Reliable Motion Estimation , 2010, ECCV.

[30]  Lei Zhang,et al.  Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising , 2016, IEEE Transactions on Image Processing.

[31]  Shie Mannor,et al.  Online PCA for Contaminated Data , 2013, NIPS.

[32]  Oscar C. Au,et al.  Temporal Video Denoising Based on Multihypothesis Motion Compensation , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[33]  Zuowei Shen,et al.  Robust video denoising using low rank matrix completion , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[34]  M. Omair Ahmad,et al.  Video Denoising Using Motion Compensated 3-D Wavelet Transform With Integrated Recursive Temporal Filtering , 2010, IEEE Transactions on Circuits and Systems for Video Technology.

[35]  Alessandro Foi,et al.  Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering , 2007, IEEE Transactions on Image Processing.

[36]  M. Omair Ahmad,et al.  Video Denoising Based on Inter-frame Statistical Modeling of Wavelet Coefficients , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[37]  Martin Kleinsteuber,et al.  Robust PCA and subspace tracking from incomplete observations using $$\ell _0$$ℓ0-surrogates , 2012, Comput. Stat..

[38]  Yoram Bresler,et al.  Learning Sparsifying Transforms , 2013, IEEE Transactions on Signal Processing.

[39]  Constantine Caramanis,et al.  Fast Algorithms for Robust PCA via Gradient Descent , 2016, NIPS.

[40]  Namrata Vaswani,et al.  An Online Algorithm for Separating Sparse and Low-Dimensional Signal Sequences From Their Sum , 2013, IEEE Transactions on Signal Processing.

[41]  Namrata Vaswani,et al.  Video denoising via online sparse and low-rank matrix decomposition , 2016, 2016 IEEE Statistical Signal Processing Workshop (SSP).

[42]  Yoram Bresler,et al.  Video denoising by online 3D sparsifying transform learning , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[43]  Karen O. Egiazarian,et al.  Video denoising by sparse 3D transform-domain collaborative filtering , 2007, 2007 15th European Signal Processing Conference.