Low-Light Image and Video Enhancement Using Deep Learning: A Survey.

Low-light image enhancement (LLIE) aims at improving the perception or interpretability of an image captured in an environment with poor illumination. Recent advances in this area are dominated by deep learning-based solutions, where many learning strategies, network structures, loss functions, training data, etc. have been employed. In this paper, we provide a comprehensive survey to cover various aspects ranging from algorithm taxonomy to unsolved open issues. To examine the generalization of existing methods, we propose a low-light image and video dataset, in which the images and videos are taken by different mobile phones' cameras under diverse illumination conditions. Besides, for the first time, we provide a unified online platform that covers many popular LLIE methods, of which the results can be produced through a user-friendly web interface. In addition to qualitative and quantitative evaluation of existing methods on publicly available and our proposed datasets, we also validate their performance in face detection in the dark. This survey together with the proposed dataset and online platform could serve as a reference source for future study and promote the development of this research field. The proposed platform and dataset as well as the collected methods, datasets, and evaluation metrics are publicly available.

[1]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[2]  Chao Dong,et al.  Recovering Realistic Texture in Image Super-Resolution by Deep Spatial Feature Transform , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[3]  Soumik Sarkar,et al.  LLNet: A deep autoencoder approach to natural low-light image enhancement , 2015, Pattern Recognit..

[4]  Jianhua Wu,et al.  MBLLEN: Low-Light Image/Video Enhancement Using CNNs , 2018, BMVC.

[5]  Andrea Vedaldi,et al.  Deep Image Prior , 2017, International Journal of Computer Vision.

[6]  Xiaojie Guo,et al.  Kindling the Darkness: A Practical Low-light Image Enhancer , 2019, ACM Multimedia.

[7]  Minh N. Do,et al.  Seeing Motion in the Dark , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[8]  Alan C. Bovik,et al.  Making a “Completely Blind” Image Quality Analyzer , 2013, IEEE Signal Processing Letters.

[9]  Lei Zhang,et al.  Blind Face Restoration via Deep Multi-scale Component Dictionaries , 2020, ECCV.

[10]  Yue Wang,et al.  From Fidelity to Perceptual Quality: A Semi-Supervised Approach for Low-Light Image Enhancement , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Chengjin Zhang,et al.  Denoising Convolutional Neural Network , 2015, 2015 IEEE International Conference on Information and Automation.

[12]  Xiangyu Xu,et al.  GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Wenhan Yang,et al.  Band Representation-Based Semi-Supervised Low-Light Image Enhancement: Bridging the Gap Between Signal Fidelity and Perceptual Quality , 2021, IEEE Transactions on Image Processing.

[14]  Jia Xu,et al.  Fast Image Processing with Fully-Convolutional Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[15]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[16]  Wonjun Kim,et al.  DSLR: Deep Stacked Laplacian Restorer for Low-Light Image Enhancement , 2021, IEEE Transactions on Multimedia.

[17]  Kede Ma,et al.  Perceptual Quality Assessment of Smartphone Photography , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Shuicheng Yan,et al.  Deep Joint Rain Detection and Removal from a Single Image , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Jian Yang,et al.  DSFD: Dual Shot Face Detector , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[21]  Jia Xu,et al.  Learning to See in the Dark , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[22]  O. A. K. Reddy,et al.  Power-Constrained Contrast Enhancement for Emissive Displays Based on Histogram Equalization , 2013 .

[23]  Delu Zeng,et al.  Removing Rain from Single Images via a Deep Detail Network , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Bo Liu,et al.  Fast Enhancement for Non-Uniform Illumination Images using Light-weight CNNs , 2020, ACM Multimedia.

[25]  Juncheng Li,et al.  Luminance-Aware Pyramid Network for Low-Light Image Enhancement , 2021, IEEE Transactions on Multimedia.

[26]  E H Land,et al.  An alternative technique for the computation of the designator in the retinex theory of color vision. , 1986, Proceedings of the National Academy of Sciences of the United States of America.

[27]  Wen Gao,et al.  Pre-Trained Image Processing Transformer , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Jonathan T. Barron,et al.  Deep bilateral learning for real-time image enhancement , 2017, ACM Trans. Graph..

[29]  Xiaoou Tang,et al.  Aesthetic-Driven Image Enhancement by Adversarial Learning , 2017, ACM Multimedia.

[30]  Alexei A. Efros,et al.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[31]  Wei Chen,et al.  EEMEFN: Low-Light Image Enhancement via Edge-Enhanced Multi-Exposure Fusion Network , 2020, AAAI.

[32]  Chi-Wing Fu,et al.  Underexposed Photo Enhancement Using Deep Illumination Estimation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Jiaying Liu,et al.  UG2 Track 2: A Collective Benchmark Effort for Evaluating and Advancing Image Understanding in Poor Visibility Environments , 2019, CVPR 2019.

[34]  Zairui Gao,et al.  An Experiment-Based Review of Low-Light Image Enhancement Methods , 2020, IEEE Access.

[35]  Yu Li,et al.  LIME: Low-Light Image Enhancement via Illumination Map Estimation , 2017, IEEE Transactions on Image Processing.

[36]  Hai-Miao Hu,et al.  Naturalness Preserved Enhancement Algorithm for Non-Uniform Illumination Images , 2013, IEEE Transactions on Image Processing.

[37]  Wenhan Yang,et al.  Integrating Semantic Segmentation and Retinex Model for Low-Light Image Enhancement , 2020, ACM Multimedia.

[38]  Xiao-Ping Zhang,et al.  A Probabilistic Method for Image Enhancement With Simultaneous Illumination and Reflectance Estimation , 2015, IEEE Transactions on Image Processing.

[39]  Jan Kautz,et al.  Loss Functions for Image Restoration With Neural Networks , 2017, IEEE Transactions on Computational Imaging.

[40]  Jean-Michel Morel,et al.  Model-Blind Video Denoising via Frame-To-Frame Training , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Joonki Paik,et al.  Low-light image enhancement using variational optimization-based Retinex model , 2017, 2017 IEEE International Conference on Consumer Electronics (ICCE).

[42]  Sam Kwong,et al.  Zero-Reference Deep Curve Estimation for Low-Light Image Enhancement , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  Ke Xu,et al.  Learning to Restore Low-Light Images via Decomposition-and-Enhancement , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Jiajun Wu,et al.  Video Enhancement with Task-Oriented Flow , 2018, International Journal of Computer Vision.

[45]  Steven McDonagh,et al.  Low Light Video Enhancement Using Synthetic Data Produced with an Intermediate Domain Mapping , 2020, ECCV.

[46]  Li Fei-Fei,et al.  Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[47]  M. Ali Akber Dewan,et al.  A Dynamic Histogram Equalization for Image Contrast Enhancement , 2007, IEEE Transactions on Consumer Electronics.

[48]  Jian Sun,et al.  Guided Image Filtering , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[49]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[50]  Sam Kwong,et al.  Towards Unsupervised Deep Image Enhancement With Generative Adversarial Network , 2020, IEEE Transactions on Image Processing.

[51]  Changhu Wang,et al.  Improving Convolutional Networks With Self-Calibrated Convolutions , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[52]  Yicong Zhou,et al.  Zero-Shot Restoration of Underexposed Images via Robust Retinex Decomposition , 2020, 2020 IEEE International Conference on Multimedia and Expo (ICME).

[53]  Wenhan Yang,et al.  Sparse Gradient Regularized Deep Retinex Network for Robust Low-Light Image Enhancement , 2021, IEEE Transactions on Image Processing.

[54]  Wan-Chi Siu,et al.  Lightening Network for Low-Light Image Enhancement , 2020, IEEE Transactions on Image Processing.

[55]  Shaodi You,et al.  Learning Temporal Consistency for Low Light Video Enhancement from Single Images , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[56]  Chen Change Loy,et al.  Learning to Enhance Low-Light Image via Zero-Reference Deep Curve Estimation , 2021, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[57]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[58]  Chen Change Loy,et al.  BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[59]  Yu Qiao,et al.  ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks , 2018, ECCV Workshops.

[60]  Peyman Milanfar,et al.  NIMA: Neural Image Assessment , 2017, IEEE Transactions on Image Processing.

[61]  Yochai Blau,et al.  The Perception-Distortion Tradeoff , 2017, CVPR.

[62]  Michael Felsberg,et al.  ECO: Efficient Convolution Operators for Tracking , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[63]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[64]  Lei Zhang,et al.  Learning a Deep Single Image Contrast Enhancer from Multi-Exposure Images , 2018, IEEE Transactions on Image Processing.

[65]  Georg Heigold,et al.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , 2021, ICLR.

[66]  Ding Liu,et al.  EnlightenGAN: Deep Light Enhancement Without Paired Supervision , 2019, IEEE Transactions on Image Processing.

[67]  Lei Zhang,et al.  Learning Image-Adaptive 3D Lookup Tables for High Performance Photo Enhancement in Real-Time , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[68]  Chee Seng Chan,et al.  Getting to Know Low-light Images with The Exclusively Dark Dataset , 2018, Comput. Vis. Image Underst..

[69]  Yung-Yu Chuang,et al.  Deep Photo Enhancer: Unpaired Learning for Image Enhancement from Photographs with GANs , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[70]  Xiaoyan Sun,et al.  Structure-Revealing Low-Light Image Enhancement Via Robust Retinex Model , 2018, IEEE Transactions on Image Processing.

[71]  Fatih Murat Porikli,et al.  LightenNet: A Convolutional Neural Network for weakly illuminated image enhancement , 2018, Pattern Recognit. Lett..

[72]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[73]  Bangshu Xiong,et al.  RetinexDIP: A Unified Deep Framework for Low-Light Image Enhancement , 2022, IEEE Transactions on Circuits and Systems for Video Technology.

[74]  Haidi Ibrahim,et al.  Brightness Preserving Dynamic Histogram Equalization for Image Contrast Enhancement , 2007, IEEE Transactions on Consumer Electronics.

[75]  Li Fei-Fei,et al.  Progressive Neural Architecture Search , 2017, ECCV.

[76]  Zhiwei Xiong,et al.  Progressive Retinex: Mutually Reinforced Illumination-Noise Perception Network for Low-Light Image Enhancement , 2019, ACM Multimedia.

[77]  Wenhan Yang,et al.  Benchmarking Low-Light Image Enhancement and Beyond , 2021, International Journal of Computer Vision.

[78]  Shuo Yang,et al.  WIDER FACE: A Face Detection Benchmark , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[79]  Junping Du,et al.  Low-Light Image Enhancement via a Deep Hybrid Network , 2019, IEEE Transactions on Image Processing.

[80]  Chul Lee,et al.  Contrast Enhancement Based on Layered Difference Representation of 2D Histograms , 2013, IEEE Transactions on Image Processing.

[81]  Kun Lu,et al.  TBEFN: A Two-Branch Exposure-Fusion Network for Low-Light Image Enhancement , 2021, IEEE Transactions on Multimedia.

[82]  Guixu Zhang,et al.  A Novel Retinex-Based Fractional-Order Variational Model for Images With Severely Low Light , 2019, IEEE Transactions on Image Processing.

[83]  Zia-ur Rahman,et al.  Properties and performance of a center/surround retinex , 1997, IEEE Trans. Image Process..

[84]  Ying Shen,et al.  Zero-Shot Restoration of Back-lit Images Using Deep Internal Learning , 2019, ACM Multimedia.

[85]  Chen Wei,et al.  Deep Retinex Decomposition for Low-Light Enhancement , 2018, BMVC.

[86]  H. Sebastian Seung,et al.  Natural Image Denoising with Convolutional Networks , 2008, NIPS.

[87]  Meng Wang,et al.  Low-Light Image Enhancement With Semi-Decoupled Decomposition , 2020, IEEE Transactions on Multimedia.

[88]  Yizhou Yu,et al.  Automatic Photo Adjustment Using Deep Neural Networks , 2014, ACM Trans. Graph..

[89]  Risheng Liu,et al.  Retinex-inspired Unrolling with Cooperative Prior Architecture Search for Low-light Image Enhancement , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[90]  Zheng-Jun Zha,et al.  Successive Graph Convolutional Network for Image De-raining , 2021, International Journal of Computer Vision.

[91]  Deli Zhao,et al.  DeepExposure: Learning to Expose Photos with Asynchronously Reinforced Adversarial Learning , 2018, NeurIPS.

[92]  Wen-Huang Cheng,et al.  LR3M: Robust Low-Light Enhancement via Low-Rank Regularized Retinex Model , 2020, IEEE Transactions on Image Processing.

[93]  Shangchen Zhou,et al.  Flexible Piecewise Curves Estimation for Photo Enhancement , 2020, ArXiv.

[94]  Jean Ponce,et al.  Learning a convolutional neural network for non-uniform motion blur removal , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[95]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[96]  Xiaojie Guo,et al.  Beyond Brightening Low-light Images , 2021, International Journal of Computer Vision.

[97]  Yinqiang Zheng,et al.  Learning to See Moving Objects in the Dark , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[98]  Zhen Hua,et al.  Low-Light Image Enhancement via Progressive-Recursive Network , 2021, IEEE Transactions on Circuits and Systems for Video Technology.

[99]  Hao He,et al.  Exposure , 2017, ACM Trans. Graph..

[100]  Bernhard Schölkopf,et al.  Online Video Deblurring via Dynamic Temporal Blending Network , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).