Lighting the Darkness in the Deep Learning Era

Low-light image enhancement (LLIE) aims at improving the perception or interpretability of an image captured in an environment with poor illumination. Recent advances in this area are dominated by deep learning-based solutions, where many learning strategies, network structures, loss functions, training data, etc. have been employed. In this paper, we provide a comprehensive survey to cover various aspects ranging from algorithm taxonomy to unsolved open issues. To examine the generalization of existing methods, we propose a large-scale low-light image and video dataset, in which the images and videos are taken by different mobile phones’ cameras under diverse illumination conditions. Besides, for the first time, we provide a unified online platform that covers many popular LLIE methods, of which the results can be produced through a user-friendly web interface. In addition to qualitative and quantitative evaluation of existing methods on publicly available and our proposed datasets, we also validate their performance in face detection in the dark. This survey together with the proposed dataset and online platform could serve as a reference source for future study and promote the development of this research field. The proposed platform and the collected methods, datasets, and evaluation metrics are publicly available and will be regularly updated at https://github.com/Li-Chongyi/Lighting-the-Darkness-in-the-Deep-Learning-Era-Open. We will release our low-light image and video dataset.

[1]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Jan Kautz,et al.  Loss Functions for Image Restoration With Neural Networks , 2017, IEEE Transactions on Computational Imaging.

[3]  O. A. K. Reddy,et al.  Power-Constrained Contrast Enhancement for Emissive Displays Based on Histogram Equalization , 2013 .

[4]  Bo Liu,et al.  Fast Enhancement for Non-Uniform Illumination Images using Light-weight CNNs , 2020, ACM Multimedia.

[5]  Sam Kwong,et al.  Towards Unsupervised Deep Image Enhancement With Generative Adversarial Network , 2020, IEEE Transactions on Image Processing.

[6]  Wen Gao,et al.  Pre-Trained Image Processing Transformer , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Juncheng Li,et al.  Luminance-Aware Pyramid Network for Low-Light Image Enhancement , 2021, IEEE Transactions on Multimedia.

[8]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[9]  Chao Dong,et al.  Recovering Realistic Texture in Image Super-Resolution by Deep Spatial Feature Transform , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[10]  Deli Zhao,et al.  DeepExposure: Learning to Expose Photos with Asynchronously Reinforced Adversarial Learning , 2018, NeurIPS.

[11]  Wen-Huang Cheng,et al.  LR3M: Robust Low-Light Enhancement via Low-Rank Regularized Retinex Model , 2020, IEEE Transactions on Image Processing.

[12]  Peyman Milanfar,et al.  NIMA: Neural Image Assessment , 2017, IEEE Transactions on Image Processing.

[13]  Yochai Blau,et al.  The Perception-Distortion Tradeoff , 2017, CVPR.

[14]  Jianhua Wu,et al.  MBLLEN: Low-Light Image/Video Enhancement Using CNNs , 2018, BMVC.

[15]  Alexei A. Efros,et al.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[16]  Zairui Gao,et al.  An Experiment-Based Review of Low-Light Image Enhancement Methods , 2020, IEEE Access.

[17]  Xiao-Ping Zhang,et al.  A Probabilistic Method for Image Enhancement With Simultaneous Illumination and Reflectance Estimation , 2015, IEEE Transactions on Image Processing.

[18]  Wei Chen,et al.  EEMEFN: Low-Light Image Enhancement via Edge-Enhanced Multi-Exposure Fusion Network , 2020, AAAI.

[19]  Chi-Wing Fu,et al.  Underexposed Photo Enhancement Using Deep Illumination Estimation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Jiaying Liu,et al.  UG2 Track 2: A Collective Benchmark Effort for Evaluating and Advancing Image Understanding in Poor Visibility Environments , 2019, CVPR 2019.

[21]  Shangchen Zhou,et al.  Flexible Piecewise Curves Estimation for Photo Enhancement , 2020, ArXiv.

[22]  Jean Ponce,et al.  Learning a convolutional neural network for non-uniform motion blur removal , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Jonathan T. Barron,et al.  Deep bilateral learning for real-time image enhancement , 2017, ACM Trans. Graph..

[24]  Joonki Paik,et al.  Low-light image enhancement using variational optimization-based Retinex model , 2017, 2017 IEEE International Conference on Consumer Electronics (ICCE).

[25]  Sam Kwong,et al.  Zero-Reference Deep Curve Estimation for Low-Light Image Enhancement , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Wan-Chi Siu,et al.  Lightening Network for Low-Light Image Enhancement , 2020, IEEE Transactions on Image Processing.

[27]  E H Land,et al.  An alternative technique for the computation of the designator in the retinex theory of color vision. , 1986, Proceedings of the National Academy of Sciences of the United States of America.

[28]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[29]  Lei Zhang,et al.  Learning a Deep Single Image Contrast Enhancer from Multi-Exposure Images , 2018, IEEE Transactions on Image Processing.

[30]  Yung-Yu Chuang,et al.  Deep Photo Enhancer: Unpaired Learning for Image Enhancement from Photographs with GANs , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[31]  Chen Change Loy,et al.  Learning to Enhance Low-Light Image via Zero-Reference Deep Curve Estimation , 2021, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Shuo Yang,et al.  WIDER FACE: A Face Detection Benchmark , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Xiaoou Tang,et al.  Aesthetic-Driven Image Enhancement by Adversarial Learning , 2017, ACM Multimedia.

[34]  Shuicheng Yan,et al.  Deep Joint Rain Detection and Removal from a Single Image , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Jia Xu,et al.  Learning to See in the Dark , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[36]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[37]  Xiaojie Guo,et al.  Beyond Brightening Low-light Images , 2021, International Journal of Computer Vision.

[38]  Jiajun Wu,et al.  Video Enhancement with Task-Oriented Flow , 2018, International Journal of Computer Vision.

[39]  Steven McDonagh,et al.  Low Light Video Enhancement Using Synthetic Data Produced with an Intermediate Domain Mapping , 2020, ECCV.

[40]  Junping Du,et al.  Low-Light Image Enhancement via a Deep Hybrid Network , 2019, IEEE Transactions on Image Processing.

[41]  Chul Lee,et al.  Contrast Enhancement Based on Layered Difference Representation of 2D Histograms , 2013, IEEE Transactions on Image Processing.

[42]  Soumik Sarkar,et al.  LLNet: A deep autoencoder approach to natural low-light image enhancement , 2015, Pattern Recognit..

[43]  Hai-Miao Hu,et al.  Naturalness Preserved Enhancement Algorithm for Non-Uniform Illumination Images , 2013, IEEE Transactions on Image Processing.

[44]  Kun Lu,et al.  TBEFN: A Two-Branch Exposure-Fusion Network for Low-Light Image Enhancement , 2021, IEEE Transactions on Multimedia.

[45]  Jean-Michel Morel,et al.  Model-Blind Video Denoising via Frame-To-Frame Training , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  Haidi Ibrahim,et al.  Brightness Preserving Dynamic Histogram Equalization for Image Contrast Enhancement , 2007, IEEE Transactions on Consumer Electronics.

[47]  Yinqiang Zheng,et al.  Learning to See Moving Objects in the Dark , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[48]  Ke Xu,et al.  Learning to Restore Low-Light Images via Decomposition-and-Enhancement , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[49]  Georg Heigold,et al.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , 2021, ICLR.

[50]  Xiaoyan Sun,et al.  Structure-Revealing Low-Light Image Enhancement Via Robust Retinex Model , 2018, IEEE Transactions on Image Processing.

[51]  Ding Liu,et al.  EnlightenGAN: Deep Light Enhancement Without Paired Supervision , 2019, IEEE Transactions on Image Processing.

[52]  Chen Change Loy,et al.  BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[53]  Lei Zhang,et al.  Learning Image-Adaptive 3D Lookup Tables for High Performance Photo Enhancement in Real-Time , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[54]  M. Ali Akber Dewan,et al.  A Dynamic Histogram Equalization for Image Contrast Enhancement , 2007, IEEE Transactions on Consumer Electronics.

[55]  Jian Sun,et al.  Guided Image Filtering , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[56]  Fatih Murat Porikli,et al.  LightenNet: A Convolutional Neural Network for weakly illuminated image enhancement , 2018, Pattern Recognit. Lett..

[57]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[58]  Changhu Wang,et al.  Improving Convolutional Networks With Self-Calibrated Convolutions , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[59]  Yicong Zhou,et al.  Zero-Shot Restoration of Underexposed Images via Robust Retinex Decomposition , 2020, 2020 IEEE International Conference on Multimedia and Expo (ICME).

[60]  Chengjin Zhang,et al.  Denoising Convolutional Neural Network , 2015, 2015 IEEE International Conference on Information and Automation.

[61]  Jia Xu,et al.  Fast Image Processing with Fully-Convolutional Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[62]  Li Fei-Fei,et al.  Progressive Neural Architecture Search , 2017, ECCV.

[63]  Zhiwei Xiong,et al.  Progressive Retinex: Mutually Reinforced Illumination-Noise Perception Network for Low-Light Image Enhancement , 2019, ACM Multimedia.

[64]  Wenhan Yang,et al.  Benchmarking Low-Light Image Enhancement and Beyond , 2021, International Journal of Computer Vision.

[65]  Li Fei-Fei,et al.  Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[66]  Chen Wei,et al.  Deep Retinex Decomposition for Low-Light Enhancement , 2018, BMVC.

[67]  H. Sebastian Seung,et al.  Natural Image Denoising with Convolutional Networks , 2008, NIPS.

[68]  Meng Wang,et al.  Low-Light Image Enhancement With Semi-Decoupled Decomposition , 2020, IEEE Transactions on Multimedia.

[69]  Yizhou Yu,et al.  Automatic Photo Adjustment Using Deep Neural Networks , 2014, ACM Trans. Graph..

[70]  Minh N. Do,et al.  Seeing Motion in the Dark , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[71]  Alan C. Bovik,et al.  Making a “Completely Blind” Image Quality Analyzer , 2013, IEEE Signal Processing Letters.

[72]  Lei Zhang,et al.  Blind Face Restoration via Deep Multi-scale Component Dictionaries , 2020, ECCV.

[73]  Wonjun Kim,et al.  DSLR: Deep Stacked Laplacian Restorer for Low-Light Image Enhancement , 2021, IEEE Transactions on Multimedia.

[74]  Yue Wang,et al.  From Fidelity to Perceptual Quality: A Semi-Supervised Approach for Low-Light Image Enhancement , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[75]  Xiangyu Xu,et al.  GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[76]  Jian Yang,et al.  DSFD: Dual Shot Face Detector , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[77]  Delu Zeng,et al.  Removing Rain from Single Images via a Deep Detail Network , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[78]  Michael Felsberg,et al.  ECO: Efficient Convolution Operators for Tracking , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[79]  Chee Seng Chan,et al.  Getting to Know Low-light Images with The Exclusively Dark Dataset , 2018, Comput. Vis. Image Underst..

[80]  Yu Li,et al.  LIME: Low-Light Image Enhancement via Illumination Map Estimation , 2017, IEEE Transactions on Image Processing.

[81]  Wenhan Yang,et al.  Integrating Semantic Segmentation and Retinex Model for Low-Light Image Enhancement , 2020, ACM Multimedia.

[82]  Guixu Zhang,et al.  A Novel Retinex-Based Fractional-Order Variational Model for Images With Severely Low Light , 2019, IEEE Transactions on Image Processing.

[83]  Zia-ur Rahman,et al.  Properties and performance of a center/surround retinex , 1997, IEEE Trans. Image Process..

[84]  Ying Shen,et al.  Zero-Shot Restoration of Back-lit Images Using Deep Internal Learning , 2019, ACM Multimedia.

[85]  Xiaojie Guo,et al.  Kindling the Darkness: A Practical Low-light Image Enhancer , 2019, ACM Multimedia.

[86]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[87]  Kede Ma,et al.  Perceptual Quality Assessment of Smartphone Photography , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[88]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[89]  Hao He,et al.  Exposure , 2017, ACM Trans. Graph..

[90]  Bernhard Schölkopf,et al.  Online Video Deblurring via Dynamic Temporal Blending Network , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).