BLNet: A Fast Deep Learning Framework for Low-Light Image Enhancement with Noise Removal and Color Restoration

Images obtained in real-world low-light conditions are not only low in brightness, but they also suffer from many other types of degradation, such as color bias, unknown noise, detail loss and halo artifacts. In this paper, we propose a very fast deep learning framework called Bringing the Lightness (denoted as BLNet) that consists of two U-Nets with a series of welldesigned loss functions to tackle all of the above degradations. Based on Retinex Theory, the decomposition net in our model can decompose low-light images into reflectance and illumination and remove noise in the reflectance during the decomposition phase. We propose a Noise and Color Bias Control module (NCBC Module) that contains a convolutional neural network and two loss functions (noise loss and color loss). This module is only used to calculate the loss functions during the training phase, so our method is very fast during the test phase. This module can smooth the reflectance to achieve the purpose of noise removal while preserving details and edge information and controlling color bias. We propose a network that can be trained to learn the mapping between low-light and normallight illumination and enhance the brightness of images taken in low-light illumination. We train and evaluate the performance of our proposed model over the real-world Low-Light (LOL) dataset), and we also test our model over several other frequently used datasets (LIME, DICM and MEF datasets). We conduct extensive experiments to demonstrate that our approach achieves a promising effect with good rubustness and generalization and outperforms many other state-of-the-art methods qualitatively and quantitatively. Our method achieves high speed because we use loss functions instead of introducing additional denoisers for noise removal and color correction. The code and model are available at https://github.com/weixinxu666/BLNet.

[1]  Wen-Huang Cheng,et al.  LR3M: Robust Low-Light Enhancement via Low-Rank Regularized Retinex Model , 2020, IEEE Transactions on Image Processing.

[2]  Yongwei Nie,et al.  Dual Illumination Estimation for Robust Exposure Correction , 2019, Comput. Graph. Forum.

[3]  Li Tao,et al.  LLCNN: A convolutional neural network for low-light image enhancement , 2017, 2017 IEEE Visual Communications and Image Processing (VCIP).

[4]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[5]  Xiaojie Guo,et al.  Kindling the Darkness: A Practical Low-light Image Enhancer , 2019, ACM Multimedia.

[6]  Wencheng Wu,et al.  The CIEDE2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations , 2005 .

[7]  Xiaoyan Sun,et al.  Structure-Revealing Low-Light Image Enhancement Via Robust Retinex Model , 2018, IEEE Transactions on Image Processing.

[8]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[9]  Yu Zhang,et al.  Self-supervised Image Enhancement Network: Training with Low Light Images Only , 2020, ArXiv.

[10]  Akira Taguchi,et al.  Color image contrast enhacement method based on differential intensity/saturation gray-levels histograms , 2013, 2013 International Symposium on Intelligent Signal Processing and Communication Systems.

[11]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[12]  Zia-ur Rahman,et al.  Properties and performance of a center/surround retinex , 1997, IEEE Trans. Image Process..

[13]  Ross B. Girshick,et al.  Mask R-CNN , 2017, 1703.06870.

[14]  Zia-ur Rahman,et al.  A multiscale retinex for bridging the gap between color images and the human observation of scenes , 1997, IEEE Trans. Image Process..

[15]  Lei Zhang,et al.  Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising , 2016, IEEE Transactions on Image Processing.

[16]  Jianhua Wu,et al.  MBLLEN: Low-Light Image/Video Enhancement Using CNNs , 2018, BMVC.

[17]  Jia Xu,et al.  Learning to See in the Dark , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[18]  Hai-Miao Hu,et al.  Naturalness Preserved Enhancement Algorithm for Non-Uniform Illumination Images , 2013, IEEE Transactions on Image Processing.

[19]  Keith E. Muller,et al.  Contrast-limited adaptive histogram equalization: speed and effectiveness , 1990, [1990] Proceedings of the First Conference on Visualization in Biomedical Computing.

[20]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[21]  Soumik Sarkar,et al.  LLNet: A deep autoencoder approach to natural low-light image enhancement , 2015, Pattern Recognit..

[22]  Delu Zeng,et al.  A fusion-based enhancing method for weakly illuminated images , 2016, Signal Process..

[23]  Chul Lee,et al.  Contrast Enhancement Based on Layered Difference Representation of 2D Histograms , 2013, IEEE Transactions on Image Processing.

[24]  Karen O. Egiazarian,et al.  Image denoising with block-matching and 3D filtering , 2006, Electronic Imaging.

[25]  Ding Liu,et al.  EnlightenGAN: Deep Light Enhancement Without Paired Supervision , 2019, IEEE Transactions on Image Processing.

[26]  Jan Kautz,et al.  Loss Functions for Image Restoration With Neural Networks , 2017, IEEE Transactions on Computational Imaging.

[27]  John D. Austin,et al.  Adaptive histogram equalization and its variations , 1987 .

[28]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[29]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Jiebo Luo,et al.  Unsupervised Real-world Low-light Image Enhancement with Decoupled Networks , 2020, ArXiv.

[31]  A. Bovik,et al.  A universal image quality index , 2002, IEEE Signal Processing Letters.

[32]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Wen Gao,et al.  A Bio-Inspired Multi-Exposure Fusion Framework for Low-light Image Enhancement , 2017, ArXiv.

[34]  Alexei A. Efros,et al.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[35]  Xiao-Ping Zhang,et al.  A Weighted Variational Model for Simultaneous Reflectance and Illumination Estimation , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Wen-Huang Cheng,et al.  Joint Enhancement and Denoising Method via Sequential Decomposition , 2018, 2018 IEEE International Symposium on Circuits and Systems (ISCAS).

[37]  Sam Kwong,et al.  Zero-Reference Deep Curve Estimation for Low-Light Image Enhancement , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Jie Ma,et al.  MSR-net: Low-light Image Enhancement Using Deep Convolutional Network , 2017, ArXiv.

[39]  David Zhang,et al.  FSIM: A Feature Similarity Index for Image Quality Assessment , 2011, IEEE Transactions on Image Processing.

[40]  Bangshu Xiong,et al.  RetinexDIP: A Unified Deep Framework for Low-Light Image Enhancement , 2022, IEEE Transactions on Circuits and Systems for Video Technology.

[41]  Haidi Ibrahim,et al.  Brightness Preserving Dynamic Histogram Equalization for Image Contrast Enhancement , 2007, IEEE Transactions on Consumer Electronics.

[42]  Chen Wei,et al.  GLADNet: Low-Light Enhancement Network with Global Awareness , 2018, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[43]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[44]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[45]  Wangmeng Zuo,et al.  Toward Convolutional Blind Denoising of Real Photographs , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  Hong Zhu,et al.  Low-Illumination Image Enhancement Algorithm Based on a Physical Lighting Model , 2019, IEEE Transactions on Circuits and Systems for Video Technology.

[47]  Jiangtao Wen,et al.  Fast efficient algorithm for enhancement of low lighting video , 2010, 2011 IEEE International Conference on Multimedia and Expo.

[48]  Ronggang Wang,et al.  A New Low-Light Image Enhancement Algorithm Using Camera Response Model , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[49]  Chen Wei,et al.  Deep Retinex Decomposition for Low-Light Enhancement , 2018, BMVC.

[50]  Kai Zeng,et al.  Perceptual Quality Assessment for Multi-Exposure Image Fusion , 2015, IEEE Transactions on Image Processing.

[51]  Xiaogang Wang,et al.  Pyramid Scene Parsing Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[52]  Guisik Kim,et al.  Low-Lightgan: Low-Light Enhancement Via Advanced Generative Adversarial Network With Task-Driven Training , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[53]  Weimin Tan,et al.  RDGAN: Retinex Decomposition Based Adversarial Learning for Low-Light Enhancement , 2019, 2019 IEEE International Conference on Multimedia and Expo (ICME).

[54]  Ge Li,et al.  LECARM: Low-Light Image Enhancement Using the Camera Response Model , 2019, IEEE Transactions on Circuits and Systems for Video Technology.

[55]  Graham D. Finlayson,et al.  Re-evaluating colour constancy algorithms , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[56]  Zhen Hua,et al.  Low-Light Image Enhancement via Progressive-Recursive Network , 2021, IEEE Transactions on Circuits and Systems for Video Technology.

[57]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[58]  Yu Li,et al.  LIME: Low-Light Image Enhancement via Illumination Map Estimation , 2017, IEEE Transactions on Image Processing.

[59]  E. Land The retinex theory of color vision. , 1977, Scientific American.

[60]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[61]  Ling Shao,et al.  STAR: A Structure and Texture Aware Retinex Model , 2019, IEEE Transactions on Image Processing.