Multi-frame GAN-based machine learning image restoration for degraded visual environments

Although single-frame machine learning image restoration techniques have been shown to be effective, the proposed multi-frame approach takes advantage of both spatial and temporal information to resolve high-resolution and high-dynamic-range images. The proposed algorithm is an extension of the previously proposed algorithm DeblurGAN-C and aims to further improve the capabilities of image restoration in degraded visual environments. The main contributions of the proposed techniques include: 1) Development of an effective framework to generate a multi-frame training dataset typical of degraded visual environments; 2) Adopting a multi-frame image restoration framework that generates a single restored image as the output; 3) Conducting substantial experiments against the generated multi-frame training dataset and demonstrate the effectiveness of the proposed multi-frame image enhancement algorithm.

[1]  Weilin Hou,et al.  Compressive line sensing imaging system in a controlled hybrid scattering environment , 2019, Optical Engineering.

[2]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  George Biddell Airy,et al.  On the diffraction of an object-glass with circular aperture , 1835 .

[4]  D. Renshaw,et al.  A Comprehensive Tool for Modeling CMOS Image-Sensor-Noise Performance , 2007, IEEE Transactions on Electron Devices.

[5]  Degradation Model Learning for Real-World Single Image Super-Resolution , 2020, ACCV.

[6]  Weifeng Liu,et al.  Correntropy: A Localized Similarity Measure , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[7]  Weilin Hou,et al.  Why does the Secchi disk disappear? An imaging perspective. , 2007, Optics express.

[8]  Marc Levoy,et al.  Handheld multi-frame super-resolution , 2019, ACM Trans. Graph..

[9]  N. Thürey,et al.  Temporally Coherent GANs for Video Super-Resolution (TecoGAN) , 2018, ArXiv.

[10]  Moon Gi Kang,et al.  Super-resolution image reconstruction: a technical overview , 2003, IEEE Signal Process. Mag..

[11]  Fraser Dalgleish,et al.  Extended-Range Undersea Laser Imaging: Current Research Status and a Glimpse at Future Technologies , 2013 .

[12]  Franck Marchis,et al.  AIDA: an adaptive image deconvolution algorithm with application to multi-frame and three-dimensional data. , 2007, Journal of the Optical Society of America. A, Optics, image science, and vision.

[13]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Mohinder Malhotra.  Single Image Haze Removal Using Dark Channel Prior , 2016 .

[15]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Bing Ouyang,et al.  DeblurGAN-C: image restoration using GAN and a correntropy based loss function in degraded visual environments , 2020, Defense + Commercial Sensing.

[17]  Ramkumar Narayanswamy,et al.  Simulating multi-camera imaging systems for depth estimation, enhanced photography and video effects , 2015 .

[18]  Luc Van Gool,et al.  NTIRE 2018 Challenge on Single Image Super-Resolution: Methods and Results , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[19]  Shuang Zhang,et al.  GAN Based Image Deblurring Using Dark Channel Prior , 2019, Computational Imaging.

[20]  Jiri Matas,et al.  DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[21]  Keigo Hirakawa,et al.  Approximations to camera sensor noise , 2013, Electronic Imaging.

[22]  Tong Li,et al.  Conditional Wasserstein generative adversarial network-gradient penalty-based approach to alleviating imbalanced data classification , 2020, Inf. Sci..