Manifold-valued Image Generation with Wasserstein Adversarial Networks

Unsupervised image generation has recently received an increasing amount of attention thanks to the great success of generative adversarial networks (GANs), particularly Wasserstein GANs. Inspired by the paradigm of real-valued image generation, this paper makes the first attempt to formulate the problem of generating manifold-valued images, which are frequently encountered in real-world applications. For the study, we specially exploit three typical manifold-valued image generation tasks: hue-saturation-value (HSV) color image generation, chromaticity-brightness (CB) color image generation, and diffusion-tensor (DT) image generation. In order to produce such kinds of images as realistic as possible, we generalize the state-of-the-art technique of Wasserstein GANs to the manifold context with exploiting Riemannian geometry. For the proposed manifold-valued image generation problem, we recommend three benchmark datasets that are CIFAR-10 HSV/CB color images, ImageNet HSV/CB color images, UCL DT image datasets. On the three datasets, we experimentally demonstrate the proposed manifold-aware Wasserestein GAN can generate high quality manifold-valued images.

[1]  A. Figalli,et al.  Optimal transportation on non-compact manifolds , 2007, 0711.4519.

[2]  Mario Lucic,et al.  Are GANs Created Equal? A Large-Scale Study , 2017, NeurIPS.

[3]  Jyothisha J. Nair,et al.  Denoising of DT-MR Images with an Iterative PCA , 2015 .

[4]  Thomas Brox,et al.  PDEs for Tensor Image Processing , 2006, Visualization and Processing of Tensor Fields.

[5]  Tomas Pfister,et al.  Learning from Simulated and Unsupervised Images through Adversarial Training , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Florence Tupin,et al.  NL-InSAR: Nonlocal Interferogram Estimation , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[7]  Ronny Bergmann,et al.  Nonlocal Inpainting of Manifold-Valued Data on Finite Weighted Graphs , 2017, GSI.

[8]  Mila Nikolova,et al.  A Nonlocal Denoising Algorithm for Manifold-Valued Images Using Second Order Statistics , 2016, SIAM J. Imaging Sci..

[9]  Yann LeCun,et al.  Energy-based Generative Adversarial Network , 2016, ICLR.

[10]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[11]  Ronny Bergmann,et al.  A Second-Order TV-Type Approach for Inpainting and Denoising Higher Dimensional Combined Cyclic and Vector Space Data , 2015, Journal of Mathematical Imaging and Vision.

[12]  Aditya Tatu,et al.  Active Contour Models for Manifold Valued Image Segmentation , 2013, Journal of Mathematical Imaging and Vision.

[13]  Yuichi Yoshida,et al.  Spectral Normalization for Generative Adversarial Networks , 2018, ICLR.

[14]  Sepp Hochreiter,et al.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium , 2017, NIPS.

[15]  Koray Kavukcuoglu,et al.  Pixel Recurrent Neural Networks , 2016, ICML.

[16]  Joachim Weickert,et al.  Median Filtering of Tensor-Valued Images , 2003, DAGM-Symposium.

[17]  Xavier Pennec,et al.  A Riemannian Framework for Tensor Computing , 2005, International Journal of Computer Vision.

[18]  Gabriele Steidl,et al.  A Second Order Nonsmooth Variational Model for Restoring Manifold-Valued Images , 2015, SIAM J. Sci. Comput..

[19]  Bernhard Schmitzer,et al.  Optimal Transport for Manifold-Valued Images , 2017, SSVM.

[20]  Léon Bottou,et al.  Wasserstein Generative Adversarial Networks , 2017, ICML.

[21]  P. Rosen,et al.  SYNTHETIC APERTURE RADAR INTERFEROMETRY TO MEASURE EARTH'S SURFACE TOPOGRAPHY AND ITS DEFORMATION , 2000 .

[22]  N. Ayache,et al.  Log‐Euclidean metrics for fast and simple calculus on diffusion tensors , 2006, Magnetic resonance in medicine.

[23]  Luc Van Gool,et al.  A Riemannian Network for SPD Matrix Learning , 2016, AAAI.

[24]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[25]  Shiguang Shan,et al.  Log-Euclidean Metric Learning on Symmetric Positive Definite Manifold with Application to Image Set Classification , 2015, ICML.

[26]  R. Bhattacharya,et al.  Nonparametic estimation of location and dispersion on Riemannian manifolds , 2002 .

[27]  R. Chan,et al.  Restoration of Manifold-Valued Images by Half-Quadratic Minimization , 2015, 1505.07029.

[28]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[29]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.

[30]  Bernhard Burgeth,et al.  Restoration of matrix fields by second-order cone programming , 2007, Computing.

[31]  Daniel C. Alexander,et al.  Camino: Open-Source Diffusion-MRI Reconstruction and Processing , 2006 .

[32]  A. Lynn Abbott,et al.  Dimensionality Reduction and Clustering on Statistical Manifolds , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[33]  Shing-Tung Yau,et al.  A Geometric View of Optimal Transportation and Generative Model , 2017, Comput. Aided Geom. Des..

[34]  G. Loeper.  Regularity of Optimal Maps on the Sphere: the Quadratic Cost and the Reflector Antenna , 2013, 1301.6229.

[35]  Alessio Figalli,et al.  The Monge–Ampère equation and its link to optimal transportation , 2013, 1310.6167.

[36]  Xiang Wei,et al.  Improving the Improved Training of Wasserstein GANs: A Consistency Term and Its Dual Effect , 2018, ICLR.

[37]  Ronny Bergmann,et al.  Inpainting of Cyclic Data Using First and Second Order Differences , 2014, EMMCVPR.

[38]  Raymond Y. K. Lau,et al.  Least Squares Generative Adversarial Networks , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[39]  Gabriele Steidl,et al.  Second Order Differences of Cyclic Data and Applications in Variational Denoising , 2014, SIAM J. Imaging Sci..