Fast Underwater Image Enhancement for Improved Visual Perception

In this letter, we present a conditional generative adversarial network-based model for real-time underwater image enhancement. To supervise the adversarial training, we formulate an objective function that evaluates the perceptual image quality based on its global content, color, local texture, and style information. We also present EUVP, a large-scale dataset of a paired and an unpaired collection of underwater images (of ‘poor’ and ‘good’ quality) that are captured using seven different cameras over various visibility conditions during oceanic explorations and human-robot collaborative experiments. In addition, we perform several qualitative and quantitative evaluations which suggest that the proposed model can learn to enhance underwater image quality from both paired and unpaired training. More importantly, the enhanced images provide improved performances of standard models for underwater object detection, human pose estimation, and saliency prediction. These results validate that it is suitable for real-time preprocessing in the autonomy pipeline by visually-guided underwater robots. The model and associated training pipelines are available at https://github.com/xahidbuffon/funie-gan.

[1]  Ming Zhu,et al.  Real-World Underwater Enhancement: Challenges, Benchmarks, and Solutions Under Natural Light , 2019, IEEE Transactions on Circuits and Systems for Video Technology.

[2]  Yann LeCun,et al.  Energy-based Generative Adversarial Network , 2016, ICLR.

[3]  Stefan B. Williams,et al.  True Color Correction of Autonomous Underwater Vehicle Imagery , 2016, J. Field Robotics.

[4]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[5]  Bin Sheng,et al.  Deep Colorization , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[6]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Md Jahidul Islam,et al.  Understanding human motion and gestures for underwater human–robot collaboration , 2018, J. Field Robotics.

[8]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[9]  Andrew Hogue,et al.  AQUA: An Amphibious Autonomous Robot , 2007, Computer.

[10]  Junaed Sattar,et al.  Toward a Generic Diver-Following Algorithm: Balancing Robustness and Efficiency in Deep Visual Detection , 2018, IEEE Robotics and Automation Letters.

[11]  Yann LeCun,et al.  Energy-based Generative Adversarial Networks , 2016, ICLR.

[12]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[13]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[14]  Zia-ur Rahman,et al.  Retinex processing for automatic image enhancement , 2002, IS&T/SPIE Electronic Imaging.

[15]  Pavel Zemcík,et al.  Compression Artifacts Removal Using Convolutional Neural Networks , 2016, J. WSCG.

[16]  Xin Fan,et al.  Real-world Underwater Enhancement: Challenging, Benchmark and Efficient Solutions , 2019, ArXiv.

[17]  Hanumant Singh,et al.  Robotic tools for deep water archaeology: Surveying an ancient shipwreck with an autonomous underwater vehicle , 2010, J. Field Robotics.

[18]  Alexei A. Efros,et al.  Colorful Image Colorization , 2016, ECCV.

[19]  Luc Van Gool,et al.  DSLR-Quality Photos on Mobile Devices with Deep Convolutional Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[20]  Shuicheng Yan,et al.  Joint Rain Detection and Removal via Iterative Region Dependent Multi-Task Learning , 2016, ArXiv.

[21]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[22]  Ting Wang,et al.  Underwater image enhancement via extended multi-scale Retinex , 2017, Neurocomputing.

[23]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[24]  Leon A. Gatys,et al.  Image Style Transfer Using Convolutional Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Jie Li,et al.  WaterGAN: Unsupervised Generative Network to Enable Real-Time Color Correction of Monocular Underwater Images , 2017, IEEE Robotics and Automation Letters.

[26]  Derya Akkaynak,et al.  A Revised Underwater Image Formation Model , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[27]  Raymond Y. K. Lau,et al.  Least Squares Generative Adversarial Networks , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[28]  Gregory Dudek,et al.  Multi-domain monitoring of marine environments using a heterogeneous robot team , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[29]  Yunchao Wei,et al.  Perceptual Generative Adversarial Networks for Small Object Detection , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Simon Osindero,et al.  Conditional Generative Adversarial Nets , 2014, ArXiv.

[31]  Djemel Ziou,et al.  Image Quality Metrics: PSNR vs. SSIM , 2010, 2010 20th International Conference on Pattern Recognition.

[32]  Léon Bottou,et al.  Wasserstein Generative Adversarial Networks , 2017, ICML.

[33]  Andrew L. Maas Rectifier Nonlinearities Improve Neural Network Acoustic Models , 2013 .

[34]  Michal Irani,et al.  Motion Analysis for Image Enhancement: Resolution, Occlusion, and Transparency , 1993, J. Vis. Commun. Image Represent..

[35]  Guoyu Wang,et al.  Underwater Image Enhancement With a Deep Residual Framework , 2019, IEEE Access.

[36]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[37]  Ali Borji,et al.  Salient Object Detection Driven by Fixation Prediction , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[38]  Hanyu Li,et al.  Underwater Image Enhancement Using a Multiscale Dense Generative Adversarial Network , 2020, IEEE Journal of Oceanic Engineering.

[39]  Mohinder Malhotra Single Image Haze Removal Using Dark Channel Prior , 2016 .

[40]  Huimin Lu,et al.  Underwater image enhancement using guided trigonometric bilateral filter and fast automatic color correction , 2013, 2013 IEEE International Conference on Image Processing.

[41]  Xiaoli Yu,et al.  Underwater-GAN: Underwater Image Restoration via Conditional Generative Adversarial Network , 2018, CVAUI/IWCF/MIPPSNA@ICPR.

[42]  Yung-Yu Chuang,et al.  Deep Photo Enhancer: Unpaired Learning for Image Enhancement from Photographs with GANs , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[43]  Xiaoou Tang,et al.  Learning a Deep Convolutional Network for Image Super-Resolution , 2014, ECCV.

[44]  Md Jahidul Islam,et al.  Enhancing Underwater Imagery Using Generative Adversarial Networks , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[45]  Shai Avidan,et al.  Underwater Single Image Color Restoration Using Haze-Lines and a New Quantitative Dataset , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[46]  Chen Gao,et al.  Human-Visual-System-Inspired Underwater Image Quality Measures , 2016, IEEE Journal of Oceanic Engineering.

[47]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[48]  Yaser Sheikh,et al.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[49]  Jinyong Jeong,et al.  Model-Assisted Multiband Fusion for Single Image Enhancement and Applications to Robot Vision , 2018, IEEE Robotics and Automation Letters.

[50]  Dacheng Tao,et al.  DehazeNet: An End-to-End System for Single Image Haze Removal , 2016, IEEE Transactions on Image Processing.

[51]  R. Steneck,et al.  Coral Reefs Under Rapid Climate Change and Ocean Acidification , 2007, Science.

[52]  Ping Tan,et al.  DualGAN: Unsupervised Dual Learning for Image-to-Image Translation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).