Single Image Deraining via Recurrent Hierarchy Enhancement Network

Single image deraining is an important problem in many computer vision tasks since rain streaks can severely hamper and degrade the visibility of images. In this paper, we propose a novel network named Recurrent Hierarchy Enhancement Network (ReHEN) to remove rain streaks from rainy images stage by stage. Unlike previous deep convolutional network methods, we adopt a Hierarchy Enhancement Unit (HEU) to fully extract local hierarchical features and generate effective features. Then a Recurrent Enhancement Unit (REU) is added to keep the useful information from HEU and benefit the rain removal in the later stages. To focus on different scales, shapes, and densities of rain streaks adaptively, Squeeze-and-Excitation (SE) block is applied in both HEU and REU to assign different scale factors to high-level features. Experiments on five synthetic datasets and a real-world rainy image set show that the proposed method outperforms the state-of-the-art methods considerably. The source code is available at https://github.com/nnUyi/ReHEN.

[1]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Wojciech Zaremba,et al.  Recurrent Neural Network Regularization , 2014, ArXiv.

[3]  Qi Xie,et al.  Should We Encode Rain Streaks in Video as Deterministic or Stochastic? , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[4]  Huafeng Wu,et al.  Residual-Guide Network for Single Image Deraining , 2018, ACM Multimedia.

[5]  Wenhan Yang,et al.  Erase or Fill? Deep Joint Recurrent Rain Removal and Reconstruction in Videos , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[6]  Delu Zeng,et al.  Removing Rain from Single Images via a Deep Detail Network , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Jie Chen,et al.  Robust Video Content Alignment and Compensation for Rain Removal in a CNN Framework , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[8]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Yu Luo,et al.  Removing Rain from a Single Image via Discriminative Sparse Coding , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[10]  Rob Fergus,et al.  Restoring an Image Taken through a Window Covered with Dirt or Rain , 2013, 2013 IEEE International Conference on Computer Vision.

[11]  Hongbin Zha,et al.  Recurrent Squeeze-and-Excitation Context Aggregation Net for Single Image Deraining , 2018, ECCV.

[12]  Danilo P. Mandic,et al.  Recurrent Neural Networks for Prediction: Learning Algorithms, Architectures and Stability , 2001 .

[13]  Shuicheng Yan,et al.  Joint Rain Detection and Removal via Iterative Region Dependent Multi-Task Learning , 2016, ArXiv.

[14]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[15]  Lei Zhang,et al.  Joint Convolutional Analysis and Synthesis Sparse Representation for Single Image Layer Separation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[16]  Alexei A. Efros,et al.  Context Encoders: Feature Learning by Inpainting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Takeo Kanade,et al.  Analysis of Rain and Snow in Frequency Space , 2008, International Journal of Computer Vision.

[18]  Jiandong Tian,et al.  Video Desnowing and Deraining Based on Matrix Decomposition , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Chang-Su Kim,et al.  Video Deraining and Desnowing Using Temporal Correlation and Low-Rank Matrix Completion , 2015, IEEE Transactions on Image Processing.

[20]  Mohammed Ghanbari,et al.  Scope of validity of PSNR in image/video quality assessment , 2008 .

[21]  Danilo P. Mandic,et al.  Recurrent Neural Networks for Prediction: Learning Algorithms, Architectures and Stability , 2001 .

[22]  Shree K. Nayar,et al.  Detection and removal of rain from videos , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[23]  Chiou-Ting Hsu,et al.  A Generalized Low-Rank Appearance Model for Spatio-temporally Correlated Rain Streaks , 2013, 2013 IEEE International Conference on Computer Vision.

[24]  Gang Sun,et al.  Squeeze-and-Excitation Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[25]  Jing Tao,et al.  Video Rain Streak Removal by Multiscale Convolutional Sparse Coding , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[26]  Vijayan K. Asari,et al.  Utilizing Local Phase Information to Remove Rain from Video , 2014, International Journal of Computer Vision.

[27]  Ting-Zhu Huang,et al.  FastDeRain: A Novel Video Rain Streak Removal Method Using Directional Gradient Priors , 2018, IEEE Transactions on Image Processing.

[28]  Dani Lischinski,et al.  Joint Bi-layer Optimization for Single-Image Rain Streak Removal , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[29]  Shuicheng Yan,et al.  Deep Joint Rain Detection and Removal from a Single Image , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Vishal M. Patel,et al.  Image De-Raining Using a Conditional Generative Adversarial Network , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[31]  Yu-Hsiang Fu,et al.  Automatic Single-Image-Based Rain Streaks Removal via Image Decomposition , 2012, IEEE Transactions on Image Processing.

[32]  Vishal M. Patel,et al.  Density-Aware Single Image De-raining Using a Multi-stream Dense Network , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[33]  Michael S. Brown,et al.  Rain Streak Removal Using Layer Priors , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Ting-Zhu Huang,et al.  A directional global sparse model for single image rain removal , 2018, Applied Mathematical Modelling.

[35]  Loong Fah Cheong,et al.  Single Image Deraining using Scale-Aware Multi-Stage Recurrent Network , 2017, ArXiv.

[36]  Huchuan Lu,et al.  Learning Dual Convolutional Neural Networks for Low-Level Vision , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[37]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[38]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.