ATFaceGAN: Single Face Semantic Aware Image Restoration and Recognition From Atmospheric Turbulence

Degradation due to atmospheric turbulence is common while capturing images at long ranges. To mitigate the degradation due to turbulence which includes deformation and blur, we propose a generative single frame restoration algorithm which disentangles the blur and deformation due to turbulence and reconstructs a restored image. The disentanglement is achieved by decomposing the distortion due to turbulence into blur and deformation components using a deblur generator and a deformation correction generator, respectively. Two paths of restoration are implemented to regularize the disentanglement and generate two restored images from a given degraded image. A fusion function combines the features of the restored images to reconstruct a sharp image with rich details. Adversarial and perceptual losses are added to reconstruct a sharp image and suppress the artifacts respectively. Extensive experiments demonstrate the effectiveness of the proposed restoration algorithm, which achieves satisfactory performance for face restoration and face recognition tasks.

[1]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[2]  Tianqi Chen,et al.  Empirical Evaluation of Rectified Activations in Convolutional Network , 2015, ArXiv.

[3]  Kevin R. Leonard,et al.  A data-constrained algorithm for the emulation of long-range turbulence-degraded video , 2019, Defense + Commercial Sensing.

[4]  Omkar M. Parkhi,et al.  VGGFace2: A Dataset for Recognising Faces across Pose and Age , 2017, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[5]  Mikhail A. Vorontsov,et al.  Automated video enhancement from a stream of atmospherically-distorted images: the lucky-region fusion approach , 2009, Optical Engineering + Applications.

[6]  Carlos D. Castillo,et al.  A Fast and Accurate System for Face Detection, Identification, and Verification , 2018, IEEE Transactions on Biometrics, Behavior, and Identity Science.

[7]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[8]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[9]  Yitzhak Yitzhaky,et al.  Detecting and tracking moving objects in long-distance imaging through turbulent medium. , 2014, Applied optics.

[10]  Vishal M. Patel,et al.  Deblurring Face Images Using Uncertainty Guided Multi-Stream Semantic Networks , 2019, IEEE Transactions on Image Processing.

[11]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Carlos D. Castillo,et al.  UMDFaces: An annotated face dataset for training deep networks , 2016, 2017 IEEE International Joint Conference on Biometrics (IJCB).

[13]  Stefanos Zafeiriou,et al.  Deep Face Deblurring , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[14]  Stefano Soatto,et al.  Video stabilization of atmospheric turbulence distortion , 2013 .

[15]  Bernhard Schölkopf,et al.  Efficient filter flow for space-variant multiframe blind deconvolution , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[16]  R. Hufnagel,et al.  Modulation Transfer Function Associated with Image Transmission through Turbulent Media , 1964 .

[17]  Yuan Xie,et al.  Removing Turbulence Effect via Hybrid Total Variation and Deformation-Guided Kernel Regression , 2016, IEEE Transactions on Image Processing.

[18]  Stefanos Zafeiriou,et al.  AgeDB: The First Manually Collected, In-the-Wild Age Database , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[19]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[20]  Weihong Deng,et al.  Cross-Age LFW: A Database for Studying Cross-Age Face Recognition in Unconstrained Environments , 2017, ArXiv.

[21]  Marwan Mattar,et al.  Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments , 2008 .

[22]  Weihong Deng,et al.  Cross-Pose LFW : A Database for Studying Cross-Pose Face Recognition in Unconstrained Environments , 2018 .

[23]  Lok Ming Lui,et al.  Restoration of atmospheric turbulence-distorted images via RPCA and quasiconformal maps , 2017, Inverse Problems.

[24]  Xiang Zhu,et al.  Removing Atmospheric Turbulence via Space-Invariant Deconvolution , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Jian Yang,et al.  FSRNet: End-to-End Learning Face Super-Resolution with Facial Priors , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[26]  Carlos D. Castillo,et al.  Frontal to profile face verification in the wild , 2016, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV).

[27]  Jiri Matas,et al.  DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[28]  M A Vorontsov,et al.  Anisoplanatic imaging through turbulent media: image recovery by local information fusion from a set of short-exposure images. , 2001, Journal of the Optical Society of America. A, Optics, image science, and vision.

[29]  Georgios Tzimiropoulos,et al.  Super-FAN: Integrated Facial Landmark Localization and Super-Resolution of Real-World Low Resolution Faces in Arbitrary Poses with GANs , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[30]  Stefano Soatto,et al.  Journal of Mathematical Imaging and Vision a Linear Systems Approach to Imaging through Turbulence a Linear Systems Approach to Imaging through Turbulence , 2022 .

[31]  J. Pearson,et al.  Atmospheric turbulence compensation using coherent optical adaptive techniques. , 1976, Applied optics.

[32]  Jan Kautz,et al.  Deep Semantic Face Deblurring , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[33]  Rama Chellappa,et al.  Unsupervised Domain-Specific Deblurring via Disentangled Representations , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Xin Yu,et al.  Super-Resolving Very Low-Resolution Face Images with Supplementary Attributes , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[35]  Enric Meinhardt,et al.  Implementation of the Centroid Method for the Correction of Turbulence , 2014, Image Process. Line.

[36]  Yuxiao Hu,et al.  MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition , 2016, ECCV.

[37]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[38]  Lok Ming Lui,et al.  Variational Models for Joint Subsampling and Reconstruction of Turbulence-Degraded Images , 2017, J. Sci. Comput..

[39]  Andrea Vedaldi,et al.  Instance Normalization: The Missing Ingredient for Fast Stylization , 2016, ArXiv.

[40]  B. Welsh,et al.  Imaging Through Turbulence , 1996 .

[41]  Rama Chellappa,et al.  ATFaceGAN: Single Face Image Restoration and Recognition from Atmospheric Turbulence , 2020, 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020).

[42]  M. Vorontsov,et al.  The principles of adaptive optics , 1985 .

[43]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.

[44]  Léon Bottou,et al.  Wasserstein GAN , 2017, ArXiv.

[45]  Anil K. Jain,et al.  IARPA Janus Benchmark - C: Face Dataset and Protocol , 2018, 2018 International Conference on Biometrics (ICB).

[46]  Chuan Li,et al.  Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks , 2016, ECCV.

[47]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[49]  Lok Ming Lui,et al.  Subsampled Turbulence Removal Network , 2018, Mathematics, Computation and Geometry of Data.

[50]  Yu Cheng,et al.  High Performance Large Scale Face Recognition with Multi-cognition Softmax and Feature Retrieval , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[51]  Xin Yu,et al.  Face Super-Resolution Guided by Facial Component Heatmaps , 2018, ECCV.

[52]  Rama Chellappa,et al.  Landmark Detection in Low Resolution Faces with Semi-Supervised Learning , 2019, ArXiv.

[53]  Xin Li,et al.  Simultaneous Video Stabilization and Moving Object Detection in Turbulence , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.