I Can See Clearly Now: Image Restoration via De-Raining

We present a method for improving segmentation tasks on images affected by adherent rain drops and streaks. We introduce a novel stereo dataset recorded using a system that allows one lens to be affected by real water droplets while keeping the other lens clear. We train a denoising generator using this dataset and show that it is effective at removing the effect of real water droplets, in the context of image reconstruction and road marking segmentation. To further test our de-noising approach, we describe a method of adding computer-generated adherent water droplets and streaks to any images, and use this technique as a proxy to demonstrate the effectiveness of our model in the context of general semantic segmentation. We benchmark our results using the CamVid road marking segmentation dataset, Cityscapes semantic segmentation datasets and our own realrain dataset, and show significant improvement on all tasks.

[1]  James F. Blinn,et al.  A Generalization of Algebraic Surface Drawing , 1982, TOGS.

[2]  Dinesh Manocha,et al.  Appearance-preserving simplification , 1998, SIGGRAPH.

[3]  Atsushi Yamashita,et al.  Removal of Adherent Waterdrops in Images by Using Multiple Cameras , 2002, MVA.

[4]  Atsushi Yamashita,et al.  A virtual wiper - restoration of deteriorated images by using multiple cameras , 2003, Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003) (Cat. No.03CH37453).

[5]  Atsushi Yamashita,et al.  A virtual wiper-restoration of deteriorated images by using a pan-tilt camera , 2004, IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004.

[6]  Atsushi Yamashita,et al.  Removal of adherent waterdrops from images acquired with stereo camera , 2005, 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[7]  Andreas Geiger,et al.  Video-based raindrop detection for improved image registration , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[8]  Roberto Cipolla,et al.  Semantic object classes in video: A high-definition ground truth database , 2009, Pattern Recognit. Lett..

[9]  Andreas Geiger,et al.  Realistic Modeling of Water Droplets for Monocular Adherent Raindrop Recognition Using Bézier Curves , 2010, ACCV Workshops.

[10]  Rob Fergus,et al.  Restoring an Image Taken through a Window Covered with Dirt or Rain , 2013, 2013 IEEE International Conference on Computer Vision.

[11]  Chang-Su Kim,et al.  Stereo video deraining and desnowing based on spatiotemporal frame warping , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[12]  Jie Chen,et al.  A rain pixel recovery algorithm for videos with highly dynamic scenes. , 2014, IEEE transactions on image processing : a publication of the IEEE Signal Processing Society.

[13]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[14]  Alex Graves,et al.  Recurrent Models of Visual Attention , 2014, NIPS.

[15]  Jie Chen,et al.  A Rain Pixel Recovery Algorithm for Videos With Highly Dynamic Scenes , 2014, IEEE Transactions on Image Processing.

[16]  Chang-Su Kim,et al.  Video Deraining and Desnowing Using Temporal Correlation and Low-Rank Matrix Completion , 2015, IEEE Transactions on Image Processing.

[17]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[18]  Chuan Li,et al.  Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks , 2016, ECCV.

[19]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Xiaochun Cao,et al.  Single Image Dehazing via Multi-scale Convolutional Neural Networks , 2016, ECCV.

[21]  Katsushi Ikeuchi,et al.  Waterdrop Stereo , 2016, ArXiv.

[22]  Katsushi Ikeuchi,et al.  Adherent Raindrop Modeling, Detectionand Removal in Video , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Antonio M. López,et al.  The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Sebastian Ramos,et al.  The Cityscapes Dataset for Semantic Urban Scene Understanding , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Alex Bewley,et al.  Addressing appearance change in outdoor robotics with adversarial domain adaptation , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[26]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Xinghao Ding,et al.  Clearing the Skies: A Deep Network Architecture for Single-Image Rain Removal , 2016, IEEE Transactions on Image Processing.

[28]  Alexei A. Efros,et al.  Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[29]  Paul Newman,et al.  1 year, 1000 km: The Oxford RobotCar dataset , 2017, Int. J. Robotics Res..

[30]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[31]  Wolfram Burgard,et al.  AdapNet: Adaptive semantic segmentation in adverse environmental conditions , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[32]  Luc Van Gool,et al.  Domain Adaptive Faster R-CNN for Object Detection in the Wild , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[33]  Paul Newman,et al.  Mark Yourself: Road Marking Segmentation via Weakly-Supervised Annotations from Multimodal Data , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[34]  Lars Hammarstrand,et al.  Long-Term Visual Localization Using Semantically Segmented Images , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[35]  Wenhan Yang,et al.  Attentive Generative Adversarial Network for Raindrop Removal from A Single Image , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[36]  George Papandreou,et al.  Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation , 2018, ECCV.

[37]  Torsten Sattler,et al.  Semantic Visual Localization , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[38]  Jan Kautz,et al.  High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.