Exploiting Global and Local Attentions for Heavy Rain Removal on Single Images

Heavy rain removal from a single image is the task of simultaneously eliminating rain streaks and fog, which can dramatically degrade the quality of captured images. Most existing rain removal methods do not generalize well for the heavy rain case. In this work, we propose a novel network architecture consisting of three sub-networks to remove heavy rain from a single image without estimating rain streaks and fog separately. The first sub-net, a U-net-based architecture that incorporates our Spatial Channel Attention (SCA) blocks, extracts global features that provide sufficient contextual information needed to remove atmospheric distortions caused by rain and fog. The second sub-net learns the additive residues information, which is useful in removing rain streak artifacts via our proposed Residual Inception Modules (RIM). The third sub-net, the multiplicative sub-net, adopts our Channel-attentive Inception Modules (CIM) and learns the essential brighter local features which are not effectively extracted in the SCA and additive sub-nets by modulating the local pixel intensities in the derained images. Our three clean image results are then combined via an attentive blending block to generate the final clean image. Our method with SCA, RIM, and CIM significantly outperforms the previous state-of-the-art single-image deraining methods on the synthetic datasets, shows considerably cleaner and sharper derained estimates on the real image datasets. We present extensive experiments and ablation studies supporting each of our method’s contributions on both synthetic and real image datasets. INDEX TERMS Heavy rain removal, Single image deraining, Image restoration, Deep learning.

[1]  Dani Lischinski,et al.  Joint Bi-layer Optimization for Single-Image Rain Streak Removal , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[2]  Ting-Zhu Huang,et al.  A directional global sparse model for single image rain removal , 2018, Applied Mathematical Modelling.

[3]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[5]  Vishal M. Patel,et al.  Confidence Measure Guided Single Image De-Raining , 2019, IEEE Transactions on Image Processing.

[6]  Qi Xie,et al.  A Model-Driven Deep Neural Network for Single Image Rain Removal , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Wenhan Yang,et al.  Erase or Fill? Deep Joint Recurrent Rain Removal and Reconstruction in Videos , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[8]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[9]  Vishal M. Patel,et al.  Uncertainty Guided Multi-Scale Residual Learning-Using a Cycle Spinning CNN for Single Image De-Raining , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Wenhan Yang,et al.  Self-Learning Video Rain Streak Removal: When Cyclic Consistency Meets Temporal Correspondence , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Wenhan Yang,et al.  Single Image Deraining: From Model-Based to Data-Driven and Beyond , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Lei Zhang,et al.  Joint Convolutional Analysis and Synthesis Sparse Representation for Single Image Layer Separation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[13]  Qinghua Hu,et al.  Progressive Image Deraining Networks: A Better and Simpler Baseline , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Loong Fah Cheong,et al.  Heavy Rain Image Restoration: Integrating Physics Model and Conditional Adversarial Learning , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Chang-Su Kim,et al.  Video Deraining and Desnowing Using Temporal Correlation and Low-Rank Matrix Completion , 2015, IEEE Transactions on Image Processing.

[16]  Shree K. Nayar,et al.  Detection and removal of rain from videos , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[17]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[18]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[19]  Rynson W. H. Lau,et al.  Spatial Attentive Single-Image Deraining With a High Quality Real Rain Dataset , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Oisin Mac Aodha,et al.  Unsupervised Monocular Depth Estimation with Left-Right Consistency , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Dorin Comaniciu,et al.  Kernel-Based Object Tracking , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[22]  Jiandong Tian,et al.  Video Desnowing and Deraining Based on Matrix Decomposition , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Zhiwei Li,et al.  FastDerainNet: A Deep Learning Algorithm for Single Image Deraining , 2020, IEEE Access.

[24]  Yuan Xie,et al.  Instance-Level Salient Object Segmentation , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Jing Tao,et al.  Video Rain Streak Removal by Multiscale Convolutional Sparse Coding , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[26]  Michael S. Brown,et al.  Rain Streak Removal Using Layer Priors , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Vishal M. Patel,et al.  Density-Aware Single Image De-raining Using a Multi-stream Dense Network , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[28]  Jong-Ok Kim,et al.  Progressive Rain Removal via a Recurrent Convolutional Network for Real Rain Videos , 2020, IEEE Access.

[29]  Chiou-Ting Hsu,et al.  A Generalized Low-Rank Appearance Model for Spatio-temporally Correlated Rain Streaks , 2013, 2013 IEEE International Conference on Computer Vision.

[30]  John W. Paisley,et al.  Lightweight Pyramid Networks for Image Deraining , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[31]  Hongbin Zha,et al.  Recurrent Squeeze-and-Excitation Context Aggregation Net for Single Image Deraining , 2018, ECCV.

[32]  Chen Chen,et al.  Multi-Scale Progressive Fusion Network for Single Image Deraining , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Yinglong Wang,et al.  Rethinking Image Deraining via Rain Streaks and Vapors , 2020, ECCV.

[34]  Junjun Jiang,et al.  Decomposition Makes Better Rain Removal: An Improved Attention-Guided Deraining Network , 2021, IEEE Transactions on Circuits and Systems for Video Technology.

[35]  Andreas Geiger,et al.  Computer Vision for Autonomous Vehicles: Problems, Datasets and State-of-the-Art , 2017, Found. Trends Comput. Graph. Vis..

[36]  Chi-Wing Fu,et al.  Depth-Attentional Features for Single-Image Rain Removal , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Shuicheng Yan,et al.  Deep Joint Rain Detection and Removal from a Single Image , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[39]  Chi-Wing Fu,et al.  Bidirectional Feature Pyramid Network with Recurrent Attention Residual Modules for Shadow Detection , 2018, ECCV.

[40]  Delu Zeng,et al.  Removing Rain from Single Images via a Deep Detail Network , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[42]  Mohammed Ghanbari,et al.  Scope of validity of PSNR in image/video quality assessment , 2008 .

[43]  Pheng-Ann Heng,et al.  SINet: A Scale-Insensitive Convolutional Neural Network for Fast Vehicle Detection , 2018, IEEE Transactions on Intelligent Transportation Systems.

[44]  Yu Luo,et al.  Removing Rain from a Single Image via Discriminative Sparse Coding , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[45]  Vishal M. Patel,et al.  Image De-Raining Using a Conditional Generative Adversarial Network , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[46]  Wei Liu,et al.  ParseNet: Looking Wider to See Better , 2015, ArXiv.