Single-image-based nonuniformity correction of uncooled long-wave infrared detectors: a deep-learning approach.

Fixed-pattern noise (FPN), which is caused by the nonuniform opto-electronic responses of microbolometer focal-plane-array (FPA) optoelectronics, imposes a challenging problem in infrared imaging systems. In this paper, we successfully demonstrate that a better single-image-based non-uniformity correction (NUC) operator can be directly learned from a large number of simulated training images instead of being handcrafted as before. Our proposed training scheme, which is based on convolutional neural networks (CNNs) and a column FPN simulation module, gives rise to a powerful technique to reconstruct the noise-free infrared image from its corresponding noisy observation. Specifically, a comprehensive column FPN model is utilized to depict the nonlinear characteristics of column amplifiers in the readout circuit of FPA. A large number of high-fidelity training images are simulated based on this model and the end-to-end residual deep network is capable of learning the intrinsic difference between undesirable FPN and original image details. Therefore, column FPN can be accurately estimated and further subtracted from the raw infrared images to obtain NUC results. Comparative results with state-of-the-art single-image-based NUC methods, using real-captured noisy infrared images, demonstrate that our proposed deep-learning-based approach delivers better performances of FPN removal, detail preservation, and artifact suppression.

[1]  Tae Hyun Kim,et al.  Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Xiubao Sui,et al.  Single Infrared Image Stripe Noise Removal Using Deep Convolutional Networks , 2017, IEEE Photonics Journal.

[3]  J Scott Tyo,et al.  Generalized algebraic scene-based nonuniformity correction algorithm. , 2005, Journal of the Optical Society of America. A, Optics, image science, and vision.

[4]  Xiubao Sui,et al.  Adaptive grayscale adjustment-based stripe noise removal method of single image , 2013 .

[5]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Bradley M. Ratliff,et al.  An algebraic algorithm for nonuniformity correction in focal-plane arrays. , 2002, Journal of the Optical Society of America. A, Optics, image science, and vision.

[7]  Chih-Cheng Hsieh,et al.  Focal-plane-arrays and CMOS readout techniques of infrared imaging systems , 1997, IEEE Trans. Circuits Syst. Video Technol..

[8]  Guohua Gu,et al.  Correction method for stripe nonuniformity. , 2010, Applied optics.

[9]  J Scott Tyo,et al.  Radiometrically accurate scene-based nonuniformity correction for array sensors. , 2003, Journal of the Optical Society of America. A, Optics, image science, and vision.

[10]  Wojciech Matusik,et al.  Statistics of Infrared Images , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Miguel Figueroa,et al.  Modeling and Compensating Temperature-Dependent Non-Uniformity Noise in IR Microbolometer Cameras , 2016, Sensors.

[12]  John G. Harris,et al.  Nonuniformity correction of infrared image sequences using the constant-statistics constraint , 1999, IEEE Trans. Image Process..

[13]  Hao Li,et al.  High-Resolution Image Inpainting Using Multi-scale Neural Patch Synthesis , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Amir Averbuch,et al.  Scene based non-uniformity correction in thermal images using Kalman filter , 2007, Image Vis. Comput..

[15]  Max J. Schulz,et al.  Nonuniformity correction and correctability of infrared focal plane arrays , 1995, Defense, Security, and Sensing.

[16]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[17]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[18]  Yanpeng Cao,et al.  Single-image-based solution for optics temperature-dependent nonuniformity correction in an uncooled long-wave infrared camera. , 2014, Optics letters.

[19]  Jérôme Gilles,et al.  Non-uniformity Correction of Infrared Images by Midway Equalization , 2012, Image Process. Line.

[20]  Michael Ying Yang,et al.  Spatially Adaptive Column Fixed-Pattern Noise Correction in Infrared Imaging System Using 1D Horizontal Differential Statistics , 2017, IEEE Photonics Journal.

[21]  Ju-feng Zhao,et al.  Single image stripe nonuniformity correction with gradient-constrained optimization model for infrared focal plane arrays , 2013 .

[22]  M M Hayat,et al.  Statistical algorithm for nonuniformity correction in focal-plane arrays. , 1999, Applied optics.

[23]  Michel Vilain,et al.  Uncooled amorphous silicon TEC-less 1/4 VGA IRFPA with 25 μm pixel-pitch for high volume applications , 2008, SPIE Defense + Commercial Sensing.

[24]  Yanpeng Cao,et al.  Strip non-uniformity correction in uncooled long-wave infrared focal plane array based on noise source characterization , 2015 .

[25]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[26]  Michael Ying Yang,et al.  Effective Strip Noise Removal for Low-Textured Infrared Images Based on 1-D Guided Filtering , 2016, IEEE Transactions on Circuits and Systems for Video Technology.

[27]  Xinlei Wang,et al.  A Fixed-Pattern Noise Correction Method Based on Gray Value Compensation for TDI CMOS Image Sensor , 2015, Sensors.

[28]  Russell C Hardie,et al.  Scene-based nonuniformity correction technique that exploits knowledge of the focal-plane array readout architecture. , 2005, Applied optics.

[29]  B. Münch,et al.  Stripe and ring artifact removal with combined wavelet--Fourier filtering. , 2009, Optics express.

[30]  Yang Yintang,et al.  Improvement in adaptive nonuniformity correction method with nonlinear model for infrared focal plane arrays , 2009 .

[31]  Qian Chen,et al.  Minimum mean square error method for stripe nonuniformity correction , 2011 .

[32]  E Armstrong,et al.  Scene-based nonuniformity correction with video sequences and registration. , 2000, Applied optics.

[33]  Jan Kautz,et al.  Loss Functions for Image Restoration With Neural Networks , 2017, IEEE Transactions on Computational Imaging.

[34]  Xiaoping Ye,et al.  A multi-scale non-uniformity correction method based on wavelet decomposition and guided filtering for uncooled long wave infrared camera , 2018, Signal Process. Image Commun..

[35]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Jonathan Martin Mooney,et al.  Responsivity Nonuniformity Limited Performance Of Infrared Staring Cameras , 1989 .

[37]  Majeed M Hayat,et al.  Kalman filtering for adaptive nonuniformity correction in infrared focal-plane arrays. , 2003, Journal of the Optical Society of America. A, Optics, image science, and vision.

[38]  Jérôme Gilles,et al.  ADMIRE: a locally adaptive single-image, non-uniformity correction and denoising algorithm: application to uncooled IR camera , 2012, Defense + Commercial Sensing.