Attention Guided Low-Light Image Enhancement with a Large Scale Low-Light Simulation Dataset

Low-light image enhancement is challenging in that it needs to consider not only brightness recovery but also complex issues like color distortion and noise, which usually hide in the dark. Simply adjusting the brightness of a low-light image will inevitably amplify those artifacts. To address this difficult problem, this paper proposes a novel end-to-end attention-guided method based on multi-branch convolutional neural network. To this end, we first construct a synthetic dataset with carefully designed low-light simulation strategies. The dataset is much larger and more diverse than existing ones. With the new dataset for training, our method learns two attention maps to guide the brightness enhancement and denoising tasks respectively. The first attention map distinguishes underexposed regions from well lit regions, and the second attention map distinguishes noises from real textures. With their guidance, the proposed multi-branch decomposition-and-fusion enhancement network works in an input adaptive way. Moreover, a reinforcement-net further enhances color and contrast of the output image. Extensive experiments on multiple datasets demonstrate that our method can produce high fidelity enhancement results for low-light images and outperforms the current state-of-the-art methods by a large margin both quantitatively and visually.

[1]  E. Land The retinex theory of color vision. , 1977, Scientific American.

[2]  Zia-ur Rahman,et al.  A multiscale retinex for bridging the gap between color images and the human observation of scenes , 1997, IEEE Trans. Image Process..

[3]  Zia-ur Rahman,et al.  Properties and performance of a center/surround retinex , 1997, IEEE Trans. Image Process..

[4]  Jesús Chamorro-Martínez,et al.  Diatom autofocusing in brightfield microscopy: a comparative study , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[5]  Sabine Süsstrunk,et al.  Measuring colorfulness in natural images , 2003, IS&T/SPIE Electronic Imaging.

[6]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[7]  Mongi A. Abidi,et al.  Gray-level grouping (GLG): an automatic method for optimized image contrast Enhancement-part I: the basic method , 2006, IEEE Transactions on Image Processing.

[8]  Paul Clough,et al.  The IAPR TC-12 Benchmark: A New Evaluation Resource for Visual Information Systems , 2006 .

[9]  Karen O. Egiazarian,et al.  Image denoising with block-matching and 3D filtering , 2006, Electronic Imaging.

[10]  Stanley M. Bileschi,et al.  Street Scenes: towards scene understanding in still images , 2006 .

[11]  Alan C. Bovik,et al.  Image information and visual quality , 2006, IEEE Trans. Image Process..

[12]  Jan Kautz,et al.  Exposure Fusion , 2007 .

[13]  Haidi Ibrahim,et al.  Brightness Preserving Dynamic Histogram Equalization for Image Contrast Enhancement , 2007, IEEE Transactions on Consumer Electronics.

[14]  Jian Sun,et al.  Single image haze removal using dark channel prior , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[16]  Yücel Altunbasak,et al.  A Histogram Modification Framework and Its Application for Image Contrast Enhancement , 2009, IEEE Transactions on Image Processing.

[17]  Cewu Lu,et al.  Image smoothing via L0 gradient minimization , 2011, ACM Trans. Graph..

[18]  Turgay Çelik,et al.  Contextual and Variational Contrast Enhancement , 2011, IEEE Transactions on Image Processing.

[19]  Yao Lu,et al.  Fast efficient algorithm for enhancement of low lighting video , 2011, ICME.

[20]  XuYi,et al.  Image smoothing via L0 gradient minimization , 2011 .

[21]  Rebecca Willett,et al.  Poisson Noise Reduction with Non-local PCA , 2012, Journal of Mathematical Imaging and Vision.

[22]  Chin-Chuan Han,et al.  Adaptive Multiscale Retinex for Image Contrast Enhancement , 2013, 2013 International Conference on Signal-Image Technology & Internet-Based Systems.

[23]  Zhou Wang,et al.  Objective Quality Assessment of Tone-Mapped Images , 2013, IEEE Transactions on Image Processing.

[24]  Akira Taguchi,et al.  Color image contrast enhacement method based on differential intensity/saturation gray-levels histograms , 2013, 2013 International Symposium on Intelligent Signal Processing and Communication Systems.

[25]  Hai-Miao Hu,et al.  Naturalness Preserved Enhancement Algorithm for Non-Uniform Illumination Images , 2013, IEEE Transactions on Image Processing.

[26]  Chul Lee,et al.  Contrast Enhancement Based on Layered Difference Representation of 2D Histograms , 2013, IEEE Transactions on Image Processing.

[27]  Xiaoou Tang,et al.  Learning a Deep Convolutional Network for Image Super-Resolution , 2014, ECCV.

[28]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[29]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[30]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[31]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[32]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[33]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[34]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Jan-Michael Frahm,et al.  From Dusk Till Dawn: Modeling in the Dark , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Alessandro Foi,et al.  Variance Stabilization for Noisy+Estimate Combination in Iterative Poisson Denoising , 2016, IEEE Signal Processing Letters.

[37]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[38]  Xiao-Ping Zhang,et al.  A Weighted Variational Model for Simultaneous Reflectance and Illumination Estimation , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Deqing Sun,et al.  Blind Image Deblurring Using Dark Channel Prior , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Delu Zeng,et al.  A fusion-based enhancing method for weakly illuminated images , 2016, Signal Process..

[41]  Alexei A. Efros,et al.  Colorful Image Colorization , 2016, ECCV.

[42]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[43]  Dacheng Tao,et al.  DehazeNet: An End-to-End System for Single Image Haze Removal , 2016, IEEE Transactions on Image Processing.

[44]  Hiroshi Ishikawa,et al.  Let there be color! , 2016, ACM Trans. Graph..

[45]  Lei Zhang,et al.  Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising , 2016, IEEE Transactions on Image Processing.

[46]  Soumik Sarkar,et al.  LLNet: A deep autoencoder approach to natural low-light image enhancement , 2015, Pattern Recognit..

[47]  Luc Van Gool,et al.  DSLR-Quality Photos on Mobile Devices with Deep Convolutional Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[48]  Li Tao,et al.  LLCNN: A convolutional neural network for low-light image enhancement , 2017, 2017 IEEE Visual Communications and Image Processing (VCIP).

[49]  Jie Ma,et al.  MSR-net: Low-light Image Enhancement Using Deep Convolutional Network , 2017, ArXiv.

[50]  Takayuki Hamamoto,et al.  Low-light color image enhancement via iterative noise reduction using RGB/NIR sensor , 2017, J. Electronic Imaging.

[51]  Jia Xu,et al.  Fast Image Processing with Fully-Convolutional Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[52]  Alexander M. Bronstein,et al.  Deep Convolutional Denoising of Low-Light Images , 2017, ArXiv.

[53]  Wen Gao,et al.  A Bio-Inspired Multi-Exposure Fusion Framework for Low-light Image Enhancement , 2017, ArXiv.

[54]  Ronggang Wang,et al.  A New Low-Light Image Enhancement Algorithm Using Camera Response Model , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[55]  Jonathan T. Barron,et al.  Deep bilateral learning for real-time image enhancement , 2017, ACM Trans. Graph..

[56]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[57]  Kaiming He,et al.  Mask R-CNN , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[58]  Yu Li,et al.  LIME: Low-Light Image Enhancement via Illumination Map Estimation , 2017, IEEE Transactions on Image Processing.

[59]  Jia Xu,et al.  Learning to See in the Dark , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[60]  Ming Yang,et al.  Image Blind Denoising with Generative Adversarial Network Based Noise Modeling , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[61]  Michael S. Brown,et al.  Classification-Driven Dynamic Image Enhancement , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[62]  Jianhua Wu,et al.  MBLLEN: Low-Light Image/Video Enhancement Using CNNs , 2018, BMVC.

[63]  Wen-Huang Cheng,et al.  Joint Enhancement and Denoising Method via Sequential Decomposition , 2018, 2018 IEEE International Symposium on Circuits and Systems (ISCAS).

[64]  Chen Wei,et al.  GLADNet: Low-Light Enhancement Network with Global Awareness , 2018, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[65]  Xinbo Gao,et al.  Perception-Preserving Convolutional Networks for Image Enhancement on Smartphones , 2018, ECCV Workshops.

[66]  Jiri Matas,et al.  DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[67]  Yun Fu,et al.  Image Super-Resolution Using Very Deep Residual Channel Attention Networks , 2018, ECCV.

[68]  Luc Van Gool,et al.  Model Adaptation with Synthetic and Real Data for Semantic Dense Foggy Scene Understanding , 2018, ECCV.

[69]  Luc Van Gool,et al.  WESPE: Weakly Supervised Photo Enhancer for Digital Cameras , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[70]  Alexei A. Efros,et al.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[71]  Chen Wei,et al.  Deep Retinex Decomposition for Low-Light Enhancement , 2018, BMVC.

[72]  Luc Van Gool,et al.  Fast Perceptual Image Enhancement , 2018, ECCV Workshops.

[73]  Lei Zhang,et al.  Learning a Deep Single Image Contrast Enhancer from Multi-Exposure Images , 2018, IEEE Transactions on Image Processing.

[74]  David Zhang,et al.  A Trilateral Weighted Sparse Coding Scheme for Real-World Image Denoising , 2018, ECCV.

[75]  Luc Van Gool,et al.  Dark Model Adaptation: Semantic Image Segmentation from Daytime to Nighttime , 2018 .

[76]  Luc Van Gool,et al.  Semantic Foggy Scene Understanding with Synthetic Data , 2017, International Journal of Computer Vision.

[77]  Chee Seng Chan,et al.  Getting to Know Low-light Images with The Exclusively Dark Dataset , 2018, Comput. Vis. Image Underst..

[78]  Luc Van Gool,et al.  Semantic Understanding of Foggy Scenes with Purely Synthetic Data , 2019, 2019 IEEE Intelligent Transportation Systems Conference (ITSC).

[79]  Xiaojie Guo,et al.  Kindling the Darkness: A Practical Low-light Image Enhancer , 2019, ACM Multimedia.

[80]  Luc Van Gool,et al.  Guided Curriculum Model Adaptation and Uncertainty-Aware Evaluation for Semantic Nighttime Image Segmentation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[81]  Junping Du,et al.  Low-Light Image Enhancement via a Deep Hybrid Network , 2019, IEEE Transactions on Image Processing.

[82]  Chi-Wing Fu,et al.  Underexposed Photo Enhancement Using Deep Illumination Estimation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[83]  Wangmeng Zuo,et al.  Toward Convolutional Blind Denoising of Real Photographs , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[84]  Zhiwei Xiong,et al.  Progressive Retinex: Mutually Reinforced Illumination-Noise Perception Network for Low-Light Image Enhancement , 2019, ACM Multimedia.

[85]  Dengxin Dai,et al.  Curriculum Model Adaptation with Synthetic and Real Data for Semantic Foggy Scene Understanding , 2019, International Journal of Computer Vision.

[86]  Yinqiang Zheng,et al.  An Integrated Enhancement Solution for 24-hour Colorful Imaging , 2020, AAAI.

[87]  Bo Liu,et al.  Fast Enhancement for Non-Uniform Illumination Images using Light-weight CNNs , 2020, ACM Multimedia.

[88]  Chunwei Tian,et al.  Image denoising using deep CNN with batch renormalization , 2020, Neural Networks.