LAFFNet: A Lightweight Adaptive Feature Fusion Network for Underwater Image Enhancement

Underwater image enhancement is an important low-level computer vision task for autonomous underwater vehicles and remotely operated vehicles to explore and understand the underwater environments. Recently, deep convolutional neural networks (CNNs) have been successfully used in many computer vision problems, and so does underwater image enhancement. There are many deep-learning-based methods with impressive performance for underwater image enhancement, but their memory and model parameter costs are hindrances in practical application. To address this issue, we propose a lightweight adaptive feature fusion network (LAFFNet). The model is the encoder-decoder model with multiple adaptive feature fusion (AAF) modules. AAF subsumes multiple branches with different kernel sizes to generate multi-scale feature maps. Furthermore, channel attention is used to merge these feature maps adaptively. Our method reduces the number of parameters from 2.5M to 0.15M (around 94% reduction) but outperforms state-of-the-art algorithms by extensive experiments. Furthermore, we demonstrate our LAFFNet effectively improves high-level vision tasks like salience object detection and single image depth estimation.

[1]  Dacheng Tao,et al.  An Underwater Image Enhancement Benchmark Dataset and Beyond , 2019, IEEE Transactions on Image Processing.

[2]  Jian-Jiun Ding,et al.  PMS-Net: Robust Haze Removal Based on Patch Map for Single Images , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Jianyuan Guo,et al.  GhostNet: More Features From Cheap Operations , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Mohinder Malhotra Single Image Haze Removal Using Dark Channel Prior , 2016 .

[5]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Sy-Yen Kuo,et al.  Efficient Reflection Removal Algorithm for Single Image by Pixel Compensation and Detail Reconstruction , 2018, 2018 IEEE 23rd International Conference on Digital Signal Processing (DSP).

[7]  Junaed Sattar,et al.  Simultaneous Enhancement and Super-Resolution of Underwater Imagery for Improved Visual Perception , 2020, Robotics: Science and Systems.

[8]  Laura Lindzey,et al.  The ARTEMIS under‐ice AUV docking system , 2018, J. Field Robotics.

[9]  Jian-Jiun Ding,et al.  JSTASR: Joint Size and Transparency-Aware Snow Removal Algorithm Based on Modified Partial Convolution and Veiling Effect Removal , 2020, ECCV.

[10]  Marcel Worring,et al.  DeepOpht: Medical Report Generation for Retinal Images via Deep Models and Visual Explanation , 2020, 2021 IEEE Winter Conference on Applications of Computer Vision (WACV).

[11]  Hao-Hsiang Yang,et al.  S3Net: A Single Stream Structure for Depth Guided Image Relighting , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[12]  Harshad Rai,et al.  Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks , 2018 .

[13]  Geoffrey E. Hinton,et al.  Distilling the Knowledge in a Neural Network , 2015, ArXiv.

[14]  Jian-Jiun Ding,et al.  PMHLD: Patch Map-Based Hybrid Learning DehazeNet for Single Image Haze Removal , 2020, IEEE Transactions on Image Processing.

[15]  Derya Akkaynak,et al.  Sea-Thru: A Method for Removing Water From Underwater Images , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Pamela C. Cosman,et al.  Underwater Image Restoration Based on Image Blurriness and Light Absorption , 2017, IEEE Transactions on Image Processing.

[17]  Hanumant Singh,et al.  Robotic tools for deep water archaeology: Surveying an ancient shipwreck with an autonomous underwater vehicle , 2010, J. Field Robotics.

[18]  Pamela C. Cosman,et al.  Single underwater image enhancement using depth estimation based on blurriness , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[19]  Derya Akkaynak,et al.  A Revised Underwater Image Formation Model , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[20]  Zhengyang Wang,et al.  Pixel Transposed Convolutional Networks , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Md Jahidul Islam,et al.  Fast Underwater Image Enhancement for Improved Visual Perception , 2020, IEEE Robotics and Automation Letters.

[22]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Jonathan T. Barron,et al.  A General and Adaptive Robust Loss Function , 2017, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Chen Gao,et al.  Human-Visual-System-Inspired Underwater Image Quality Measures , 2016, IEEE Journal of Oceanic Engineering.

[25]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Xiaoli Ma,et al.  When Causal Intervention Meets Adversarial Examples and Image Masking for Deep Neural Networks , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[27]  A. Cantor Optics of the atmosphere--Scattering by molecules and particles , 1978, IEEE Journal of Quantum Electronics.

[28]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[29]  Chao-Han Huck Yang,et al.  Y-Net: Multi-Scale Feature Aggregation Network With Wavelet Structure Similarity Loss Function For Single Image Dehazing , 2020, ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[30]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[32]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[33]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[34]  Sos S. Agaian,et al.  Parameterized Logarithmic Framework for Image Enhancement , 2011, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[35]  Hao-Hsiang Yang,et al.  Wavelet U-Net and the Chromatic Adaptation Transform for Single Image Dehazing , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[36]  Silvia Silva da Costa Botelho,et al.  Transmission Estimation in Underwater Single Images , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[37]  Adrian Galdran,et al.  Automatic Red-Channel underwater image restoration , 2015, J. Vis. Commun. Image Represent..

[38]  Jesper Tegnér,et al.  Synthesizing New Retinal Symptom Images by Multiple Generative Models , 2018, ACCV Workshops.

[39]  Sy-Yen Kuo,et al.  Color Channel-Based Smoke Removal Algorithm Using Machine Learning for Static Images , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[40]  Chao Gao,et al.  BASNet: Boundary-Aware Salient Object Detection , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Chin-Hui Lee,et al.  Characterizing Speech Adversarial Examples Using Self-Attention U-Net Enhancement , 2020, ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[42]  Yi Ouyang,et al.  Causal Inference Q-Network: Toward Resilient Reinforcement Learning , 2021, ArXiv.

[43]  Jan Kautz,et al.  Loss Functions for Image Restoration With Neural Networks , 2017, IEEE Transactions on Computational Imaging.

[44]  Yu-Chiang Frank Wang,et al.  Wavelet Channel Attention Module With A Fusion Network For Single Image Deraining , 2020, 2020 IEEE International Conference on Image Processing (ICIP).

[45]  Md Jahidul Islam,et al.  Enhancing Underwater Imagery Using Generative Adversarial Networks , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[46]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[47]  Chunhua Shen,et al.  Enforcing Geometric Constraints of Virtual Normal for Depth Prediction , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[48]  Hao-Hsiang Yang,et al.  Multi-modal Bifurcated Network for Depth Guided Image Relighting , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[49]  Enhua Wu,et al.  Squeeze-and-Excitation Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.