FastDeRain: A Novel Video Rain Streak Removal Method Using Directional Gradient Priors

Rain streaks removal is an important issue in outdoor vision systems and has recently been investigated extensively. In this paper, we propose a novel video rain streak removal approach FastDeRain, which fully considers the discriminative characteristics of rain streaks and the clean video in the gradient domain. Specifically, on the one hand, rain streaks are sparse and smooth along the direction of the raindrops, whereas on the other hand, clean videos exhibit piecewise smoothness along the rain-perpendicular direction and continuity along the temporal direction. Theses smoothness and continuity result in the sparse distribution in the different directional gradient domain. Thus, we minimize: 1) the <inline-formula> <tex-math notation="LaTeX">$\ell _{1}$ </tex-math></inline-formula> norm to enhance the sparsity of the underlying rain streaks; 2) two <inline-formula> <tex-math notation="LaTeX">$\ell _{1}$ </tex-math></inline-formula> norm of unidirectional total variation regularizers to guarantee the anisotropic spatial smoothness; and 3) an <inline-formula> <tex-math notation="LaTeX">$\ell _{1}$ </tex-math></inline-formula> norm of the time-directional difference operator to characterize the temporal continuity. A split augmented Lagrangian shrinkage algorithm-based algorithm is designed to solve the proposed minimization model. Experiments conducted on synthetic and real data demonstrate the effectiveness and efficiency of the proposed method. According to the comprehensive quantitative performance measures, our approach outperforms other state-of-the-art methods, especially on account of the running time. The code of FastDeRain can be downloaded at <uri>https://github.com/TaiXiangJiang/FastDeRain</uri>.

[1]  Katsushi Ikeuchi,et al.  Adherent Raindrop Modeling, Detectionand Removal in Video , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Jie Chen,et al.  A Rain Pixel Recovery Algorithm for Videos With Highly Dynamic Scenes , 2014, IEEE Transactions on Image Processing.

[3]  Vishal M. Patel,et al.  Convolutional Sparse and Low-Rank Coding-Based Rain Streak Removal , 2017, 2017 IEEE Winter Conference on Applications of Computer Vision (WACV).

[4]  Li-Wei Kang,et al.  Visual Depth Guided Color Image Rain Streaks Removal Using Sparse Coding , 2014, IEEE Transactions on Circuits and Systems for Video Technology.

[5]  Delu Zeng,et al.  Removing Rain from Single Images via a Deep Detail Network , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Subhransu Maji,et al.  Classification using intersection kernel support vector machines is efficient , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Michael K. Ng,et al.  A New Convex Optimization Model for Multiplicative Noise and Blur Removal , 2014, SIAM J. Imaging Sci..

[8]  Haojie Li,et al.  Deep Layer Prior Optimization for Single Image Rain Streaks Removal , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[9]  Xinghao Ding,et al.  Clearing the Skies: A Deep Network Architecture for Single-Image Rain Removal , 2016, IEEE Transactions on Image Processing.

[10]  Jiandong Tian,et al.  Video Desnowing and Deraining Based on Matrix Decomposition , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Ting-Zhu Huang,et al.  Matrix factorization for low-rank tensor completion using framelet prior , 2018, Inf. Sci..

[12]  A. Bovik,et al.  A universal image quality index , 2002, IEEE Signal Processing Letters.

[13]  Lei Zhang,et al.  Gradient Magnitude Similarity Deviation: A Highly Efficient Perceptual Image Quality Index , 2013, IEEE Transactions on Image Processing.

[14]  Xin Jin,et al.  Video Inpainting Based on Joint Gradient and Noise Minimization , 2016, PCM.

[15]  Dorin Comaniciu,et al.  Kernel-Based Object Tracking , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[16]  Lei Zhang,et al.  Learning Aggregated Transmission Propagation Networks for Haze Removal and Beyond , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[17]  Qi Xie,et al.  Should We Encode Rain Streaks in Video as Deterministic or Stochastic? , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[18]  Michael S. Brown,et al.  Single Image Rain Streak Decomposition Using Layer Priors , 2017, IEEE Transactions on Image Processing.

[19]  Rob Fergus,et al.  Restoring an Image Taken through a Window Covered with Dirt or Rain , 2013, 2013 IEEE International Conference on Computer Vision.

[20]  Sudipta Mukhopadhyay,et al.  Removal of rain from videos: a review , 2014, Signal Image Video Process..

[21]  Prashanth H.S.,et al.  Image Scaling Comparison Using Universal Image Quality Index , 2009, 2009 International Conference on Advances in Computing, Control, and Telecommunication Technologies.

[22]  Hongbin Zha,et al.  Recurrent Squeeze-and-Excitation Context Aggregation Net for Single Image Deraining , 2018, ECCV.

[23]  Jie Chen,et al.  Robust Video Content Alignment and Compensation for Rain Removal in a CNN Framework , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[24]  Alan C. Bovik,et al.  Image information and visual quality , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[25]  M. A. Sutton,et al.  Systematic errors in digital image correlation caused by intensity interpolation , 2000 .

[26]  Sudipta Mukhopadhyay,et al.  Video post processing: low-latency spatiotemporal approach for detection and removal of rain , 2012 .

[27]  Shree K. Nayar,et al.  Vision and Rain , 2007, International Journal of Computer Vision.

[28]  Guomin Sun,et al.  On sum and stability of g-frames in Hilbert spaces , 2018 .

[29]  Michael S. Brown,et al.  Rain Streak Removal Using Layer Priors , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Ting-Zhu Huang,et al.  A directional global sparse model for single image rain removal , 2018, Applied Mathematical Modelling.

[31]  Zhenjiang Miao,et al.  A Saliency Prior Context Model for Real-Time Object Tracking , 2017, IEEE Transactions on Multimedia.

[32]  Lei Zhang,et al.  Joint Convolutional Analysis and Synthesis Sparse Representation for Single Image Layer Separation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[33]  Huchuan Lu,et al.  Learning Dual Convolutional Neural Networks for Low-Level Vision , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[34]  Wenhan Yang,et al.  Attentive Generative Adversarial Network for Raindrop Removal from A Single Image , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[35]  Shutao Li,et al.  Hyperspectral Image Super-Resolution via Non-local Sparse Tensor Factorization , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Thomas B. Moeslund,et al.  Rain Removal in Traffic Surveillance: Does it Matter? , 2018, IEEE Transactions on Intelligent Transportation Systems.

[37]  Jun Cai,et al.  Video-Based Automatic Incident Detection for Smart Roads: The Outdoor Environmental Challenges Regarding False Alarms , 2008, IEEE Transactions on Intelligent Transportation Systems.

[38]  Shutao Li,et al.  Fusing Hyperspectral and Multispectral Images via Coupled Sparse Tensor Factorization , 2018, IEEE Transactions on Image Processing.

[39]  Shih-Chia Huang,et al.  Error-Optimized Sparse Representation for Single Image Rain Removal , 2017, IEEE Transactions on Industrial Electronics.

[40]  Zhijun Zhang,et al.  Weighted Low-Rank Tensor Recovery for Hyperspectral Image Restoration , 2017, IEEE Transactions on Cybernetics.

[41]  Yu Luo,et al.  Removing Rain from a Single Image via Discriminative Sparse Coding , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[42]  Liang Lin,et al.  Non-locally Enhanced Encoder-Decoder Network for Single Image De-raining , 2018, ACM Multimedia.

[43]  Jie Huang,et al.  Directional ℓ0 Sparse Modeling for Image Stripe Noise Removal , 2018, Remote. Sens..

[44]  Xiaojie Guo,et al.  Fast Single Image Rain Removal via a Deep Decomposition-Composition Network , 2018, CVPR 2018.

[45]  Chiou-Ting Hsu,et al.  A Generalized Low-Rank Appearance Model for Spatio-temporally Correlated Rain Streaks , 2013, 2013 IEEE International Conference on Computer Vision.

[46]  Michael K. Ng,et al.  Deblurring and Sparse Unmixing for Hyperspectral Images , 2013, IEEE Transactions on Geoscience and Remote Sensing.

[47]  Sheng Zhong,et al.  Transformed Low-Rank Model for Line Pattern Noise Removal , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[48]  Bing Zeng,et al.  A Hierarchical Approach for Rain or Snow Removing in a Single Color Image , 2017, IEEE Transactions on Image Processing.

[49]  David Zhang,et al.  FSIM: A Feature Similarity Index for Image Quality Assessment , 2011, IEEE Transactions on Image Processing.

[50]  Yi Ma,et al.  Generalized Tensor Total Variation minimization for visual data recovery? , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[51]  Vishal M. Patel,et al.  Image De-Raining Using a Conditional Generative Adversarial Network , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[52]  Shunsuke Ono,et al.  Cartoon-Texture Image Decomposition Using Blockwise Low-Rank Texture Characterization , 2014, IEEE Transactions on Image Processing.

[53]  Yu-Hsiang Fu,et al.  Automatic Single-Image-Based Rain Streaks Removal via Image Decomposition , 2012, IEEE Transactions on Image Processing.

[54]  José M. Bioucas-Dias,et al.  An Augmented Lagrangian Approach to the Constrained Optimization Formulation of Imaging Inverse Problems , 2009, IEEE Transactions on Image Processing.

[55]  Ting-Zhu Huang,et al.  A novel nonconvex approach to recover the low-tubal-rank tensor data: when t-SVD meets PSSV , 2017, ArXiv.

[56]  Xiang Zhang,et al.  A Bayesian Approach to Camouflaged Moving Object Detection , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[57]  Huafeng Wu,et al.  Residual-Guide Network for Single Image Deraining , 2018, ACM Multimedia.

[58]  Shuicheng Yan,et al.  Deep Joint Rain Detection and Removal from a Single Image , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[59]  Vishal M. Patel,et al.  Density-Aware Single Image De-raining Using a Multi-stream Dense Network , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[60]  Sheng Zhong,et al.  Remote Sensing Image Stripe Noise Removal: From Image Decomposition Perspective , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[61]  Naoto Yokoya,et al.  Nonlocal Tensor Completion for Multitemporal Remotely Sensed Images’ Inpainting , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[62]  Dani Lischinski,et al.  Joint Bi-layer Optimization for Single-Image Rain Streak Removal , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[63]  Jie Li,et al.  Single image deraining via decorrelating the rain streaks and background scene in gradient domain , 2018, Pattern Recognit..

[64]  Chang-Hwan Son,et al.  Rain removal via shrinkage of sparse codes and learned rain dictionary , 2016, 2016 IEEE International Conference on Multimedia & Expo Workshops (ICMEW).

[65]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[66]  Hai Liu,et al.  Simultaneous Destriping and Denoising for Remote Sensing Images With Unidirectional Total Variation and Sparse Representation , 2014, IEEE Geoscience and Remote Sensing Letters.

[67]  Yu-Chiang Frank Wang,et al.  Exploiting image structural similarity for single image rain removal , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[68]  Thierry Bouwmans,et al.  Traditional and recent approaches in background modeling for foreground detection: An overview , 2014, Comput. Sci. Rev..

[69]  Jenq-Neng Hwang,et al.  DesnowNet: Context-Aware Deep Network for Snow Removal , 2017, IEEE Transactions on Image Processing.

[70]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[71]  Chang-Su Kim,et al.  Video Deraining and Desnowing Using Temporal Correlation and Low-Rank Matrix Completion , 2015, IEEE Transactions on Image Processing.

[72]  Ting-Zhu Huang,et al.  A Novel Tensor-Based Video Rain Streaks Removal Approach via Utilizing Discriminatively Intrinsic Priors , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[73]  M. Werman,et al.  Simulation of Rain in Videos , 2002 .

[74]  Urbano Nunes,et al.  Trainable classifier-fusion schemes: An application to pedestrian detection , 2009, 2009 12th International IEEE Conference on Intelligent Transportation Systems.

[75]  Shree K. Nayar,et al.  Detection and removal of rain from videos , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[76]  David Zhang,et al.  Simultaneous Fidelity and Regularization Learning for Image Restoration , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[77]  Jing Tao,et al.  Video Rain Streak Removal by Multiscale Convolutional Sparse Coding , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[78]  Vijayan K. Asari,et al.  Utilizing Local Phase Information to Remove Rain from Video , 2014, International Journal of Computer Vision.

[79]  Ting-Zhu Huang,et al.  Hyperspectral image restoration using framelet-regularized low-rank nonnegative matrix factorization , 2018, Applied Mathematical Modelling.

[80]  Xiaochun Cao,et al.  Deep Video Dehazing With Semantic Segmentation , 2019, IEEE Transactions on Image Processing.

[81]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[82]  Shuai Yang,et al.  D3R-Net: Dynamic Routing Residue Recurrent Network for Video Rain Removal , 2019, IEEE Transactions on Image Processing.

[83]  Wenhan Yang,et al.  Erase or Fill? Deep Joint Recurrent Rain Removal and Reconstruction in Videos , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[84]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..