Single infrared image enhancement using a deep convolutional neural network

Abstract In this paper, we propose a deep learning method for single infrared image enhancement. A fully convolutional neural network (CNN) is used to produce images with enhanced contrast and details. The conditional generative adversarial networks are incorporated into the optimization framework to avoid the background noise being amplified and further enhance the contrast and details. The existing convolutional neural network architectures, such as residual architectures and encoder–decoder architectures, fail to achieve the best results both in terms of network performance and application scope for infrared image enhancement task. To address this problem, we specifically design a new refined convolutional neural architecture that produces visually very appealing results with higher contrast and sharper details compared to other network architectures. Visible images are used for training since there are fewer infrared images. Proper training samples are generated to ensure that the network trained on visible images can be well applied to infrared images. Experiments demonstrate that our approach outperforms existing image enhancement algorithms in terms of contrast and detail enhancement. Code is available at https://github.com/Kuangxd/IE-CGAN .

[1]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Axel-Christian Guei,et al.  Deep learning enhancement of infrared face images using generative adversarial networks. , 2018, Applied optics.

[3]  Alexei A. Efros,et al.  Context Encoders: Feature Learning by Inpainting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  M. Ravishankar,et al.  Modified Contrast Limited Adaptive Histogram Equalization Based on Local Contrast Enhancement for Mammogram Images , 2013 .

[5]  Xiubao Sui,et al.  Single Infrared Image Optical Noise Removal Using a Deep Convolutional Neural Network , 2018, IEEE Photonics Journal.

[6]  A. Rényi.  On Measures of Entropy and Information , 1961 .

[7]  Zhou Wang,et al.  Multiscale structural similarity for image quality assessment , 2003, The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003.

[8]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[9]  Dinggang Shen,et al.  3D conditional generative adversarial networks for high-quality PET image estimation at low dose , 2018, NeuroImage.

[10]  P. Shanmugavadivu,et al.  Particle swarm optimized multi-objective histogram equalization for image enhancement , 2014 .

[11]  Dimitris N. Metaxas,et al.  StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[12]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[13]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[14]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[15]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Clément Farabet,et al.  Torch7: A Matlab-like Environment for Machine Learning , 2011, NIPS 2011.

[17]  Kyoung Mu Lee,et al.  Deeply-Recursive Convolutional Network for Image Super-Resolution , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[19]  Anil A. Bharath,et al.  Task Specific Adversarial Cost Function , 2016, ArXiv.

[20]  Jitendra Malik,et al.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[21]  Namil Kim,et al.  Thermal Image Enhancement using Convolutional Neural Network , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[22]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[24]  Wojciech Zaremba,et al.  Improved Techniques for Training GANs , 2016, NIPS.

[25]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[26]  Xiubao Sui,et al.  Single Infrared Image Stripe Noise Removal Using Deep Convolutional Networks , 2017, IEEE Photonics Journal.

[27]  Kyungjae Lee,et al.  Brightness-Based Convolutional Neural Network for Thermal Image Enhancement , 2017, IEEE Access.

[28]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[29]  Zexuan Zhu,et al.  Computational intelligence in optical remote sensing image processing , 2018, Appl. Soft Comput..

[30]  Virgil E. Vickers,et al.  Plateau equalization algorithm for real-time display of high-quality infrared imagery , 1996 .

[31]  Simon Osindero,et al.  Conditional Generative Adversarial Nets , 2014, ArXiv.

[32]  Yanfei Liu,et al.  SatCNN: satellite image dataset classification using agile convolutional neural networks , 2017 .

[33]  Jian Sun,et al.  Identity Mappings in Deep Residual Networks , 2016, ECCV.

[34]  Lei Xiong,et al.  Dim infrared image enhancement based on convolutional neural network , 2018, Neurocomputing.