Manifold-Valued Image Generation with Wasserstein Generative Adversarial Nets

Generative modeling over natural images is one of the most fundamental machine learning problems. However, few modern generative models, including Wasserstein Generative Adversarial Nets (WGANs), are studied on manifold-valued images that are frequently encountered in real-world applications. To fill the gap, this paper first formulates the problem of generating manifold-valued images and exploits three typical instances: hue-saturation-value (HSV) color image generation, chromaticity-brightness (CB) color image generation, and diffusion-tensor (DT) image generation. For the proposed generative modeling problem, we then introduce a theorem of optimal transport to derive a new Wasserstein distance of data distributions on complete manifolds, enabling us to achieve a tractable objective under the WGAN framework. In addition, we recommend three benchmark datasets that are CIFAR-10 HSV/CB color images, ImageNet HSV/CB color images, UCL DT image datasets. On the three datasets, we experimentally demonstrate the proposed manifold-aware WGAN model can generate more plausible manifold-valued images than its competitors.

[1]  Yuichi Yoshida,et al.  Spectral Normalization for Generative Adversarial Networks , 2018, ICLR.

[2]  G. Loeper.  Regularity of Optimal Maps on the Sphere: the Quadratic Cost and the Reflector Antenna , 2013, 1301.6229.

[3]  Alessio Figalli,et al.  The Monge–Ampère equation and its link to optimal transportation , 2013, 1310.6167.

[4]  Xiang Wei,et al.  Improving the Improved Training of Wasserstein GANs: A Consistency Term and Its Dual Effect , 2018, ICLR.

[5]  Mario Lucic,et al.  Are GANs Created Equal? A Large-Scale Study , 2017, NeurIPS.

[6]  Ronny Bergmann,et al.  Inpainting of Cyclic Data Using First and Second Order Differences , 2014, EMMCVPR.

[7]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[8]  Mila Nikolova,et al.  A Nonlocal Denoising Algorithm for Manifold-Valued Images Using Second Order Statistics , 2016, SIAM J. Imaging Sci..

[9]  Ronny Bergmann,et al.  A Second-Order TV-Type Approach for Inpainting and Denoising Higher Dimensional Combined Cyclic and Vector Space Data , 2015, Journal of Mathematical Imaging and Vision.

[10]  Xavier Pennec,et al.  A Riemannian Framework for Tensor Computing , 2005, International Journal of Computer Vision.

[11]  Tomas Pfister,et al.  Learning from Simulated and Unsupervised Images through Adversarial Training , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  R. McCann.  Polar factorization of maps on Riemannian manifolds , 2001 .

[13]  Hongdong Li,et al.  Kernel Methods on the Riemannian Manifold of Symmetric Positive Definite Matrices , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  H. Hopf,et al.  Ueber den Begriff der vollständigen differentialgeometrischen Fläche , 1931 .

[15]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.

[16]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[17]  Jyothisha J. Nair,et al.  Denoising of DT-MR Images with an Iterative PCA , 2015 .

[18]  Raymond Y. K. Lau,et al.  Least Squares Generative Adversarial Networks , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[19]  Gabriele Steidl,et al.  Second Order Differences of Cyclic Data and Applications in Variational Denoising , 2014, SIAM J. Imaging Sci..

[20]  Shing-Tung Yau,et al.  A Geometric View of Optimal Transportation and Generative Model , 2017, Comput. Aided Geom. Des..

[21]  Koray Kavukcuoglu,et al.  Pixel Recurrent Neural Networks , 2016, ICML.

[22]  C. Villani.  Optimal Transport: Old and New , 2008 .

[23]  Léon Bottou,et al.  Wasserstein Generative Adversarial Networks , 2017, ICML.

[24]  Daniel C. Alexander,et al.  Camino: Open-Source Diffusion-MRI Reconstruction and Processing , 2006 .

[25]  A. Lynn Abbott,et al.  Dimensionality Reduction and Clustering on Statistical Manifolds , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[26]  Klaus Hahn,et al.  A review of diffusion tensor magnetic resonance imaging computational methods and software tools , 2011, Comput. Biol. Medicine.

[27]  Sepp Hochreiter,et al.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium , 2017, NIPS.

[28]  Shiguang Shan,et al.  Log-Euclidean Metric Learning on Symmetric Positive Definite Manifold with Application to Image Set Classification , 2015, ICML.

[29]  R. Bhattacharya,et al.  Nonparametic estimation of location and dispersion on Riemannian manifolds , 2002 .

[30]  Aditya Tatu,et al.  Active Contour Models for Manifold Valued Image Segmentation , 2013, Journal of Mathematical Imaging and Vision.

[31]  N. Ayache,et al.  Log‐Euclidean metrics for fast and simple calculus on diffusion tensors , 2006, Magnetic resonance in medicine.

[32]  Luc Van Gool,et al.  A Riemannian Network for SPD Matrix Learning , 2016, AAAI.

[33]  Gabriele Steidl,et al.  A Second Order Nonsmooth Variational Model for Restoring Manifold-Valued Images , 2015, SIAM J. Sci. Comput..

[34]  Bernhard Schmitzer,et al.  Optimal Transport for Manifold-Valued Images , 2017, SSVM.

[35]  A. Figalli,et al.  Optimal transportation on non-compact manifolds , 2007, 0711.4519.

[36]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[37]  Yann LeCun,et al.  Energy-based Generative Adversarial Network , 2016, ICLR.

[38]  Vittorio Murino,et al.  From Covariance Matrices to Covariance Operators: Data Representation from Finite to Infinite-Dimensional Settings , 2016 .

[39]  Nicholas Ayache,et al.  Geometric Means in a Novel Vector Space Structure on Symmetric Positive-Definite Matrices , 2007, SIAM J. Matrix Anal. Appl..