Image deraining using multi-scale aggregated generator network

Abstract. Rain streaks attached to a camera may seriously affect the visibility of the background and considerably degrade image quality. We handle this problem by removing rain streaks to convert a rainy image into a clean one. This creates a problem in that the information with respect to the background of the occluded parts is close to being lost for the most part. To remove rain streaks from a single image, we employ a multi-scale aggregated generator network. Differing from previous generative adversarial networks, an enhanced generator block (EGB) is applied in the generator; this broadens the receptive field of the input features and enhances the attention to rain streaks details. In addition, both the generator and the discriminator networks contain the visual mask that learns the rain streak regions and their surroundings during training. By putting the mask forward, the generator network focuses on rain streaks and the surrounding regions and the discriminator network is capable of evaluating the local consistency of the restored regions.

[1]  Vishal M. Patel,et al.  Convolutional Sparse and Low-Rank Coding-Based Rain Streak Removal , 2017, 2017 IEEE Winter Conference on Applications of Computer Vision (WACV).

[2]  Li-Wei Kang,et al.  Self-Learning Based Image Decomposition With Applications to Single Image Denoising , 2014, IEEE Transactions on Multimedia.

[3]  Michael S. Brown,et al.  Single Image Rain Streak Decomposition Using Layer Priors , 2017, IEEE Transactions on Image Processing.

[4]  Youzhao Yang,et al.  Single Image Deraining using a Recurrent Multi-scale Aggregation and Enhancement Network , 2019, 2019 IEEE International Conference on Multimedia and Expo (ICME).

[5]  Rob Fergus,et al.  Restoring an Image Taken through a Window Covered with Dirt or Rain , 2013, 2013 IEEE International Conference on Computer Vision.

[6]  Xiaodong Xie,et al.  FFA-Net: Feature Fusion Attention Network for Single Image Dehazing , 2019, AAAI.

[7]  Hongbin Zha,et al.  Recurrent Squeeze-and-Excitation Context Aggregation Net for Single Image Deraining , 2018, ECCV.

[8]  Minh N. Do,et al.  Semantic Image Inpainting with Deep Generative Models , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Qinghua Hu,et al.  Progressive Image Deraining Networks: A Better and Simpler Baseline , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Alex Graves,et al.  Recurrent Models of Visual Attention , 2014, NIPS.

[11]  Vishal M. Patel,et al.  Image De-Raining Using a Conditional Generative Adversarial Network , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[12]  Loong Fah Cheong,et al.  Heavy Rain Image Restoration: Integrating Physics Model and Conditional Adversarial Learning , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Ling Shao,et al.  Conditional Variational Image Deraining , 2020, IEEE Transactions on Image Processing.

[14]  Jürgen Schmidhuber,et al.  Learning to Forget: Continual Prediction with LSTM , 2000, Neural Computation.

[15]  Michael S. Brown,et al.  Rain Streak Removal Using Layer Priors , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Thomas S. Huang,et al.  Generative Image Inpainting with Contextual Attention , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[17]  Robby T. Tan,et al.  Visibility in bad weather from a single image , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[18]  Xinghao Ding,et al.  Clearing the Skies: A Deep Network Architecture for Single-Image Rain Removal , 2016, IEEE Transactions on Image Processing.

[19]  Gang Hua,et al.  Gated Context Aggregation Network for Image Dehazing and Deraining , 2018, 2019 IEEE Winter Conference on Applications of Computer Vision (WACV).

[20]  Pheng-Ann Heng,et al.  Deep Multi-Model Fusion for Single-Image Dehazing , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[21]  Yu Luo,et al.  Removing Rain from a Single Image via Discriminative Sparse Coding , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[22]  Ali Farhadi,et al.  YOLOv3: An Incremental Improvement , 2018, ArXiv.

[23]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[24]  Yu-Hsiang Fu,et al.  Automatic Single-Image-Based Rain Streaks Removal via Image Decomposition , 2012, IEEE Transactions on Image Processing.

[25]  Shuicheng Yan,et al.  Deep Joint Rain Detection and Removal from a Single Image , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Chi-Wing Fu,et al.  Depth-Attentional Features for Single-Image Rain Removal , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Dani Lischinski,et al.  Joint Bi-layer Optimization for Single-Image Rain Streak Removal , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[28]  Rynson W. H. Lau,et al.  Spatial Attentive Single-Image Deraining With a High Quality Real Rain Dataset , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Vishal M. Patel,et al.  Density-Aware Single Image De-raining Using a Multi-stream Dense Network , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[30]  Delu Zeng,et al.  Removing Rain from Single Images via a Deep Detail Network , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Alex Graves,et al.  DRAW: A Recurrent Neural Network For Image Generation , 2015, ICML.

[32]  Wenhan Yang,et al.  Attentive Generative Adversarial Network for Raindrop Removal from A Single Image , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[33]  Xiaochun Cao,et al.  Single Image Dehazing via Multi-scale Convolutional Neural Networks , 2016, ECCV.

[34]  Mohinder Malhotra.  Single Image Haze Removal Using Dark Channel Prior , 2016 .

[35]  Shree K. Nayar,et al.  Detection and removal of rain from videos , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[36]  Li-Wei Kang,et al.  Visual Depth Guided Color Image Rain Streaks Removal Using Sparse Coding , 2014, IEEE Transactions on Circuits and Systems for Video Technology.

[37]  Lei Zhang,et al.  Variational Image Deraining , 2020, 2020 IEEE Winter Conference on Applications of Computer Vision (WACV).

[38]  Chiou-Ting Hsu,et al.  A Generalized Low-Rank Appearance Model for Spatio-temporally Correlated Rain Streaks , 2013, 2013 IEEE International Conference on Computer Vision.

[39]  Ming-Hsuan Yang,et al.  Generative Face Completion , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Hiroshi Ishikawa,et al.  Globally and locally consistent image completion , 2017, ACM Trans. Graph..

[41]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[42]  Shree K. Nayar,et al.  Vision and Rain , 2007, International Journal of Computer Vision.

[43]  Bo Zhao,et al.  Diversified Visual Attention Networks for Fine-Grained Object Classification , 2016, IEEE Transactions on Multimedia.

[44]  Djemel Ziou,et al.  Image Quality Metrics: PSNR vs. SSIM , 2010, 2010 20th International Conference on Pattern Recognition.