Semi-supervised Learning with GANs: Manifold Invariance with Improved Inference

Semi-supervised learning methods using Generative Adversarial Networks (GANs) have shown promising empirical success recently. Most of these methods use a shared discriminator/classifier which discriminates real examples from fake while also predicting the class label. Motivated by the ability of the GANs generator to capture the data manifold well, we propose to estimate the tangent space to the data manifold using GANs and employ it to inject invariances into the classifier. In the process, we propose enhancements over existing methods for learning the inverse mapping (i.e., the encoder) which greatly improves in terms of semantic similarity of the reconstructed sample with the input sample. We observe considerable empirical gains in semi-supervised learning over baselines, particularly in the cases when the number of labeled examples is low. We also provide insights into how fake examples influence the semi-supervised learning procedure.

[1]  M. Spivak.  A comprehensive introduction to differential geometry , 1979 .

[2]  S. Shankar Sastry,et al.  Generalized principal component analysis (GPCA) , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Zhuowen Tu,et al.  Learning Generative Models via Discriminative Approaches , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[4]  G. Lerman,et al.  Probabilistic Recovery of Multiple Subspaces in Point Clouds by Geometric lp Minimization , 2010, 1002.1994.

[5]  Lorenzo Rosasco,et al.  Some Recent Advances in Multiscale Geometric Analysis of Point Clouds , 2011 .

[6]  Pascal Vincent,et al.  The Manifold Tangent Classifier , 2011, NIPS.

[7]  Stephen Smale,et al.  A Topological View of Unsupervised Learning from Noisy Data , 2011, SIAM J. Comput..

[8]  Alexander P. Kuleshov,et al.  Tangent Bundle Manifold Learning via Grassmann&Stiefel Eigenmaps , 2012, ArXiv.

[9]  Lorenzo Rosasco,et al.  Learning Manifolds with K-Means and K-Flats , 2012, NIPS.

[10]  Yann LeCun,et al.  Transformation Invariance in Pattern Recognition - Tangent Distance and Tangent Propagation , 2012, Neural Networks: Tricks of the Trade.

[11]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[12]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[13]  Max Welling,et al.  Semi-supervised Learning with Deep Generative Models , 2014, NIPS.

[14]  A. Bernstein.  Data-based Manifold Reconstruction via Tangent Bundle Manifold Learning , 2014 .

[15]  Lin Sun,et al.  Laplacian Auto-Encoders: An explicit learning of nonlinear data manifold , 2015, Neurocomputing.

[16]  Tapani Raiko,et al.  Semi-supervised Learning with Ladder Networks , 2015, NIPS.

[17]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[18]  Tomaso A. Poggio,et al.  Learning with Group Invariant Features: A Kernel Perspective , 2015, NIPS.

[19]  Jonathon Shlens,et al.  Explaining and Harnessing Adversarial Examples , 2014, ICLR.

[20]  Yann LeCun,et al.  Stacked What-Where Auto-encoders , 2015, ArXiv.

[21]  Shin Ishii,et al.  Distributional Smoothing with Virtual Adversarial Training , 2015, ICLR 2016.

[22]  Shakir Mohamed,et al.  Learning in Implicit Generative Models , 2016, ArXiv.

[23]  Jost Tobias Springenberg,et al.  Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks , 2015, ICLR.

[24]  Wojciech Zaremba,et al.  Improved Techniques for Training GANs , 2016, NIPS.

[25]  Ole Winther,et al.  Auxiliary Deep Generative Models , 2016, ICML.

[26]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[27]  Augustus Odena,et al.  Semi-Supervised Learning with Generative Adversarial Networks , 2016, ArXiv.

[28]  Sepp Hochreiter,et al.  Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) , 2015, ICLR.

[29]  Cheng Soon Ong,et al.  Linking losses for density ratio and class-probability estimation , 2016, ICML.

[30]  Sebastian Nowozin,et al.  f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization , 2016, NIPS.

[31]  Vaibhava Goel,et al.  McGan: Mean and Covariance Feature Matching GAN , 2017, ICML.

[32]  Trevor Darrell,et al.  Adversarial Feature Learning , 2016, ICLR.

[33]  Bernhard Schölkopf,et al.  Local Group Invariant Representations via Orbit Embeddings , 2016, AISTATS.

[34]  David Berthelot,et al.  BEGAN: Boundary Equilibrium Generative Adversarial Networks , 2017, ArXiv.

[35]  Timo Aila,et al.  Temporal Ensembling for Semi-Supervised Learning , 2016, ICLR.

[36]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.

[37]  Bowen Zhou,et al.  Learning Loss Functions for Semi-supervised Learning via Discriminative Adversarial Networks , 2017, ArXiv.

[38]  Aaron C. Courville,et al.  Adversarially Learned Inference , 2016, ICLR.