Robustness via Uncertainty-aware Cycle Consistency

Unpaired image-to-image translation refers to learning inter-image-domain mapping without corresponding image pairs. Existing methods learn deterministic mappings without explicitly modelling the robustness to outliers or predictive uncertainty, leading to performance degradation when encountering unseen perturbations at test time. To address this, we propose a novel probabilistic method based on Uncertainty-aware Generalized Adaptive Cycle Consistency (UGAC), which models the per-pixel residual by generalized Gaussian distribution, capable of modelling heavy-tailed distributions. We compare our model with a wide variety of state-of-the-art methods on various challenging tasks including unpaired image translation of natural images, using standard datasets, spanning autonomous driving, maps, facades, and also in medical imaging domain consisting of MRI. Experimental results demonstrate that our method exhibits stronger robustness towards unseen perturbations in test data. Code is released here: https: //github.com/ExplainableML/UncertaintyAwareCycleConsistency.

[1]  Klaus H. Maier-Hein,et al.  A Probabilistic U-Net for Segmentation of Ambiguous Images , 2018, NeurIPS.

[2]  Shaohua Kevin Zhou,et al.  Cross-Domain Synthesis of Medical Images Using Efficient Location-Sensitive Deep Network , 2015, MICCAI.

[3]  Raghav Mehta,et al.  Uncertainty Evaluation Metric for Brain Tumour Segmentation , 2020, ArXiv.

[4]  Maja Pantic,et al.  DINO: A Conditional Energy-Based GAN for Domain Translation , 2021, ICLR.

[5]  Trevor Hastie,et al.  Statistical Learning with Sparsity: The Lasso and Generalizations , 2015 .

[6]  Suyash P. Awate,et al.  Uncertainty-aware GAN with Adaptive Loss for Robust MRI Image Enhancement , 2021, 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW).

[7]  Sébastien Ourselin,et al.  Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks , 2018, Neurocomputing.

[8]  Daniel Rueckert,et al.  Identifying population differences in whole-brain structural networks: A machine learning approach , 2010, NeuroImage.

[9]  Zoubin Ghahramani,et al.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning , 2015, ICML.

[10]  Ken D. Sauer,et al.  A generalized Gaussian image model for edge-preserving MAP estimation , 1993, IEEE Trans. Image Process..

[11]  Tomaso A. Poggio,et al.  Fast and Flexible Inference of Joint Distributions from their Marginals , 2019, ICML.

[12]  Henkjan Huisman,et al.  Supervised Uncertainty Quantification for Segmentation with Multiple Annotations , 2019, MICCAI.

[13]  Kun Zhang,et al.  Geometry-Consistent Generative Adversarial Networks for One-Sided Unsupervised Domain Mapping , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Kilian Q. Weinberger,et al.  On Calibration of Modern Neural Networks , 2017, ICML.

[16]  Nojun Kwak,et al.  Generalized mean for robust principal component analysis , 2016, Pattern Recognit..

[17]  Michael J. Black,et al.  On the unification of line processes, outlier rejection, and robust statistics with applications in early vision , 1996, International Journal of Computer Vision.

[18]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[19]  Jaakko Lehtinen,et al.  High-Quality Self-Supervised Deep Image Denoising , 2019, NeurIPS.

[20]  Tanmoy Bhattacharya,et al.  The need for uncertainty quantification in machine-assisted medical decision making , 2019, Nat. Mach. Intell..

[21]  Mauricio Reyes,et al.  Assessing Reliability and Challenges of Uncertainty Estimations for Medical Image Segmentation , 2019, MICCAI.

[22]  Luc Van Gool,et al.  Deep Unfolding Network for Image Super-Resolution , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Lior Wolf,et al.  One-Sided Unsupervised Domain Mapping , 2017, NIPS.

[24]  P. J. Huber.  The 1972 Wald Lecture Robust Statistics: A Review , 1972 .

[25]  Saad Nadeem,et al.  Augmenting Colonoscopy Using Extended and Directional CycleGAN for Lossy Image Translation , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Bin Yang,et al.  MedGAN: Medical Image Translation using GANs , 2018, Comput. Medical Imaging Graph..

[27]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Nannan Li,et al.  MRI Cross-Modality Image-to-Image Translation , 2020, Scientific Reports.

[29]  Suyash P. Awate,et al.  Robust Super-Resolution Gan, with Manifold-Based and Perception Loss , 2019, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).

[30]  Ngai-Man Cheung,et al.  Dist-GAN: An Improved GAN Using Distance Constraints , 2018, ECCV.

[31]  Doina Precup,et al.  Exploring Uncertainty Measures in Deep Networks for Multiple Sclerosis Lesion Detection and Segmentation , 2018, MICCAI.

[32]  Suyash P. Awate,et al.  A Mixed-Supervision Multilevel GAN Framework for Image Quality Enhancement , 2019, MICCAI.

[33]  Alexei A. Efros,et al.  Contrastive Learning for Unpaired Image-to-Image Translation , 2020, ECCV.

[34]  T. Lindvall.  Lectures on the Coupling Method , 1992 .

[35]  Daniel C. Alexander,et al.  Uncertainty modelling in deep learning for safer neuroimage enhancement: Demonstration in diffusion MRI , 2020, NeuroImage.

[36]  Aykut Erdem,et al.  Image Synthesis in Multi-Contrast MRI With Conditional Generative Adversarial Networks , 2018, IEEE Transactions on Medical Imaging.

[37]  Kate Saenko,et al.  Adversarial Self-Defense for Cycle-Consistent GANs , 2019, NeurIPS.

[38]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[39]  Radim Sára,et al.  Spatial Pattern Templates for Recognition of Objects with Regular Structure , 2013, GCPR.

[40]  Kwanghoon Sohn,et al.  Memory-guided Unsupervised Image-to-image Translation , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Qingping Zhou,et al.  Bayesian Inference and Uncertainty Quantification for Medical Image Reconstruction with Poisson Data , 2020, SIAM J. Imaging Sci..

[42]  Claudio Gentile,et al.  The Robustness of the p-Norm Algorithms , 1999, COLT '99.

[43]  Jianfei Cai,et al.  The Spatially-Correlative Loss for Various Image Translation Tasks , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Jan Kautz,et al.  Unsupervised Image-to-Image Translation Networks , 2017, NIPS.

[45]  Yaxing Wang,et al.  DeepI2I: Enabling Deep Hierarchical Image-to-Image Translation by Transferring from GANs , 2020, NeurIPS.

[46]  Antonio Criminisi,et al.  Bayesian Image Quality Transfer with CNNs: Exploring Uncertainty in dMRI Super-Resolution , 2017, MICCAI.

[47]  Sebastian Ramos,et al.  The Cityscapes Dataset for Semantic Urban Scene Understanding , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  Philippe Réfrégier,et al.  Statistical Image Processing Techniques for Noisy Images: An Application-Oriented Approach , 2003 .

[49]  Gary F. Egan,et al.  Towards Lower-Dose PET using Physics-Based Uncertainty-Aware Multimodal Learning with Robustness to Out-of-Distribution Data , 2021, Medical Image Anal..

[50]  Georg Langs,et al.  Exploiting Epistemic Uncertainty of Anatomy Segmentation for Anomaly Detection in Retinal OCT , 2019, IEEE Transactions on Medical Imaging.

[51]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[52]  Charles Blundell,et al.  Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles , 2016, NIPS.

[53]  Edgar Simo-Serra,et al.  Let there be Color!: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification , 2016 .

[54]  Chuyang Ye,et al.  An improved deep network for tissue microstructure estimation with uncertainty quantification , 2020, Medical Image Anal..

[55]  Alex Kendall,et al.  What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? , 2017, NIPS.

[56]  Saeid Nahavandi,et al.  Neural Network-Based Uncertainty Quantification: A Survey of Methodologies and Applications , 2018, IEEE Access.

[57]  Eli Shechtman,et al.  Spatially-Adaptive Pixelwise Networks for Fast Image Translation , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).