Semi-Supervised Landmark-Guided Restoration of Atmospheric Turbulent Images

Image degradation due to atmospheric turbulence (AT), which is common while capturing images at long ranges, adversely affects the performance of tasks such as face alignment and face recognition. To the best of our knowledge, there does not exist any dataset consisting of turbulence-degraded face images along with their annotated landmarks and ground-truth clean images, making supervised training challenging. In this paper, we present a semisupervised method for jointly extracting facial landmarks and restoring the degraded images by exploiting the semantic information from the landmarks. The proposed approach learns to generate AT images by combining the content from a clean image and turbulence information from AT images in an unpaired manner. Next, we use heatmaps from the landmark localization network as a prior to the image restoration module. Subsequently, we impose heatmap consistency loss and heatmap confidence loss to regularize the restored images. Extensive experiments demonstrate the effectiveness of the proposed network, which achieves an NME of 2.797 on the task of landmark localization for strong turbulent images and yields improved restoration results compared to state-of-the-art methods.

[1]  Rama Chellappa,et al.  KEPLER: Keypoint and Pose Estimation of Unconstrained Faces by Learning Efficient H-CNN Regressors , 2017, 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017).

[2]  Jonathan Tompson,et al.  Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation , 2014, NIPS.

[3]  Peter V. Gehler,et al.  DeepCut: Joint Subset Partition and Labeling for Multi Person Pose Estimation , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[5]  Xiang Zhu,et al.  Removing Atmospheric Turbulence via Space-Invariant Deconvolution , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Stefano Soatto,et al.  Video stabilization of atmospheric turbulence distortion , 2013 .

[7]  Mikhail A. Vorontsov,et al.  Automated video enhancement from a stream of atmospherically-distorted images: the lucky-region fusion approach , 2009, Optical Engineering + Applications.

[8]  Jiri Matas,et al.  DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[9]  Bernt Schiele,et al.  DeeperCut: A Deeper, Stronger, and Faster Multi-person Pose Estimation Model , 2016, ECCV.

[10]  Yuan Xie,et al.  Removing Turbulence Effect via Hybrid Total Variation and Deformation-Guided Kernel Regression , 2016, IEEE Transactions on Image Processing.

[11]  Xin Yu,et al.  Super-Resolving Very Low-Resolution Face Images with Supplementary Attributes , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[12]  Lok Ming Lui,et al.  Subsampled Turbulence Removal Network , 2018, Mathematics, Computation and Geometry of Data.

[13]  Varun Ramakrishna,et al.  Convolutional Pose Machines , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Rama Chellappa,et al.  ATFaceGAN: Single Face Image Restoration and Recognition from Atmospheric Turbulence , 2020, 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020).

[15]  J. Pearson,et al.  Atmospheric turbulence compensation using coherent optical adaptive techniques. , 1976, Applied optics.

[16]  Jan Kautz,et al.  Deep Semantic Face Deblurring , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[17]  Xiaoou Tang,et al.  Facial Landmark Detection by Deep Multi-task Learning , 2014, ECCV.

[18]  Enric Meinhardt,et al.  Implementation of the Centroid Method for the Correction of Turbulence , 2014, Image Process. Line.

[19]  Stefanos Zafeiriou,et al.  300 Faces in-the-Wild Challenge: The First Facial Landmark Localization Challenge , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[20]  Andrew Zisserman,et al.  Flowing ConvNets for Human Pose Estimation in Videos , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[21]  Rama Chellappa,et al.  Unsupervised Domain-Specific Deblurring via Disentangled Representations , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  M. Vorontsov,et al.  The principles of adaptive optics , 1985 .

[23]  Carlos D. Castillo,et al.  The Do’s and Don’ts for CNN-Based Face Verification , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[24]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[25]  Lior Wolf,et al.  Unsupervised Cross-Domain Image Generation , 2016, ICLR.

[26]  Christian Szegedy,et al.  DeepPose: Human Pose Estimation via Deep Neural Networks , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[27]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[28]  Georgios Tzimiropoulos,et al.  Human Pose Estimation via Convolutional Part Heatmap Regression , 2016, ECCV.

[29]  Alexei A. Efros,et al.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[30]  Kevin R. Leonard,et al.  A data-constrained algorithm for the emulation of long-range turbulence-degraded video , 2019, Defense + Commercial Sensing.

[31]  Lok Ming Lui,et al.  Restoration of atmospheric turbulence-distorted images via RPCA and quasiconformal maps , 2017, Inverse Problems.

[32]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[33]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Dong Liu,et al.  High-Resolution Representations for Labeling Pixels and Regions , 2019, ArXiv.

[35]  Vladlen Koltun,et al.  Photographic Image Synthesis with Cascaded Refinement Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[36]  Georgios Tzimiropoulos,et al.  Super-FAN: Integrated Facial Landmark Localization and Super-Resolution of Real-World Low Resolution Faces in Arbitrary Poses with GANs , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[37]  Xiaogang Wang,et al.  Deep Convolutional Network Cascade for Facial Point Detection , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[38]  Jian Yang,et al.  FSRNet: End-to-End Learning Face Super-Resolution with Facial Priors , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[39]  Stefanos Zafeiriou,et al.  Deep Face Deblurring , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[40]  Yi Yang,et al.  Style Aggregated Network for Facial Landmark Detection , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[41]  Horst Bischof,et al.  Annotated Facial Landmarks in the Wild: A large-scale, real-world database for facial landmark localization , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[42]  Shuo Yang,et al.  WIDER FACE: A Face Detection Benchmark , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  Georgios Tzimiropoulos,et al.  How Far are We from Solving the 2D & 3D Face Alignment Problem? (and a Dataset of 230,000 3D Facial Landmarks) , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[44]  Xin Yu,et al.  Face Super-Resolution Guided by Facial Component Heatmaps , 2018, ECCV.

[45]  Rama Chellappa,et al.  Landmark Detection in Low Resolution Faces with Semi-Supervised Learning , 2019, ArXiv.

[46]  Xin Li,et al.  Simultaneous Video Stabilization and Moving Object Detection in Turbulence , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[47]  Bernhard Schölkopf,et al.  Efficient filter flow for space-variant multiframe blind deconvolution , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[48]  Dong Liu,et al.  Deep High-Resolution Representation Learning for Human Pose Estimation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[49]  M A Vorontsov,et al.  Anisoplanatic imaging through turbulent media: image recovery by local information fusion from a set of short-exposure images. , 2001, Journal of the Optical Society of America. A, Optics, image science, and vision.

[50]  Stefano Soatto,et al.  Journal of Mathematical Imaging and Vision a Linear Systems Approach to Imaging through Turbulence a Linear Systems Approach to Imaging through Turbulence , 2022 .

[51]  Lok Ming Lui,et al.  Variational Models for Joint Subsampling and Reconstruction of Turbulence-Degraded Images , 2017, J. Sci. Comput..

[52]  David Berthelot,et al.  BEGAN: Boundary Equilibrium Generative Adversarial Networks , 2017, ArXiv.