Graph Regularized Variational Ladder Networks for Semi-Supervised Learning

To tackle the problem of semi-supervised learning (SSL), we propose a new autoencoder-based deep model. Ladder networks (LN) is an autoencoder-based method for representation learning which has been successfully applied on unsupervised learning and semi-supervised learning. However, It ignores the manifold information of high-dimensional data and usually achieves unmeaning features which are very difficult to use in the subsequent tasks, such as prediction and recognition. To these issues, we proposed Graph Regularized Variational Ladder Networks (GRVLN), which explicitly and implicitly employs the manifold structure of data. Our contributions can be summarized as two folds: (1) Graph regularization is used to build all decoder layers, which explicitly promotes the manifold learning via graph laplacian matrixs; (2) Variational autoencoder is used as the backbone instead of traditional autoencoder in the encoder layers for implicitly learning the manifold structure of data distribution. Compared with ladder networks and other autoencoder-based methods, GRVLN achieves superior performance in semi-supervised classification tasks. Experimental results show that our method also has a comparable performance with state-of-the-art methods on several benchmark data sets.

[1]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[2]  Tara N. Sainath,et al.  Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups , 2012, IEEE Signal Processing Magazine.

[3]  Dacheng Tao,et al.  Deformed Graph Laplacian for Semisupervised Learning , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[4]  Mikhail Belkin,et al.  Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples , 2006, J. Mach. Learn. Res..

[5]  Wojciech Zaremba,et al.  Improved Techniques for Training GANs , 2016, NIPS.

[6]  Jianzhong Wu,et al.  Stacked Sparse Autoencoder (SSAE) for Nuclei Detection on Breast Cancer Histopathology Images , 2016, IEEE Transactions on Medical Imaging.

[7]  Zoubin Ghahramani,et al.  Combining active learning and semi-supervised learning using Gaussian fields and harmonic functions , 2003, ICML 2003.

[8]  Jost Tobias Springenberg,et al.  Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks , 2015, ICLR.

[9]  Bernhard Schölkopf,et al.  Learning with Local and Global Consistency , 2003, NIPS.

[10]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[11]  Fan Yang,et al.  Good Semi-supervised Learning That Requires a Bad GAN , 2017, NIPS.

[12]  Navdeep Jaitly,et al.  Towards End-To-End Speech Recognition with Recurrent Neural Networks , 2014, ICML.

[13]  Ruslan Salakhutdinov,et al.  Multimodal Neural Language Models , 2014, ICML.

[14]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[15]  Khaled Shaalan,et al.  Speech Recognition Using Deep Neural Networks: A Systematic Review , 2019, IEEE Access.

[16]  Honglak Lee,et al.  An Analysis of Single-Layer Networks in Unsupervised Feature Learning , 2011, AISTATS.

[17]  Hui Li,et al.  DenseFuse: A Fusion Approach to Infrared and Visible Images , 2018, IEEE Transactions on Image Processing.

[18]  Chuan Wang,et al.  Video Inpainting by Jointly Learning Temporal Structure and Spatial Details , 2018, AAAI.

[19]  Yiyi Liao,et al.  Place Classification With a Graph Regularized Deep Neural Network , 2015, IEEE Transactions on Cognitive and Developmental Systems.

[20]  Avrim Blum,et al.  The Bottleneck , 2021, Monopsony Capitalism.

[21]  Zhi-Hua Zhou,et al.  Tri-training: exploiting unlabeled data using three classifiers , 2005, IEEE Transactions on Knowledge and Data Engineering.

[22]  Xiaoning Song,et al.  SP-GAN: Self-Growing and Pruning Generative Adversarial Networks , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[23]  Qingming Huang,et al.  A Graph Regularized Deep Neural Network for Unsupervised Image Representation Learning , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Xiaojin Zhu,et al.  Semi-Supervised Learning , 2010, Encyclopedia of Machine Learning.

[25]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[26]  Hui Li,et al.  MDLatLRR: A Novel Decomposition Method for Infrared and Visible Image Fusion , 2018, IEEE Transactions on Image Processing.

[27]  Andrew Y. Ng,et al.  Reading Digits in Natural Images with Unsupervised Feature Learning , 2011 .

[28]  Yann LeCun,et al.  The mnist database of handwritten digits , 2005 .

[29]  Tom Minka,et al.  Principled Hybrids of Generative and Discriminative Models , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[30]  Jun Zhu,et al.  Triple Generative Adversarial Nets , 2017, NIPS.

[31]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[32]  Samy Bengio,et al.  Show and tell: A neural image caption generator , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Thorsten Joachims,et al.  Transductive Inference for Text Classification using Support Vector Machines , 1999, ICML.

[34]  Jason Weston,et al.  Deep learning via semi-supervised embedding , 2008, ICML '08.

[35]  Pascal Vincent,et al.  Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion , 2010, J. Mach. Learn. Res..

[36]  Daniel Cohen-Or,et al.  Global-to-local generative model for 3D shapes , 2018, ACM Trans. Graph..

[37]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[38]  Chuan Wang,et al.  Look, Listen and Learn - A Multimodal LSTM for Speaker Identification , 2016, AAAI.

[39]  Tapani Raiko,et al.  Semi-supervised Learning with Ladder Networks , 2015, NIPS.

[40]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[41]  Xiaojin Zhu,et al.  --1 CONTENTS , 2006 .

[42]  Yoshua Bengio,et al.  Large-Scale Feature Learning With Spike-and-Slab Sparse Coding , 2012, ICML.

[43]  Geoffrey Zweig,et al.  From captions to visual concepts and back , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Xiao-Jun Wu,et al.  Dual Encoder-Decoder Based Generative Adversarial Networks for Disentangled Facial Representation Learning , 2019, IEEE Access.

[45]  Josef Kittler,et al.  Semi-Supervised Learning Based on GAN With Mean and Variance Feature Matching , 2019, IEEE Transactions on Cognitive and Developmental Systems.

[46]  Xiao-Jun Wu,et al.  Discriminative Feature Learning via Sparse Autoencoders with Label Consistency Constraints , 2019, Neural Processing Letters.

[47]  Bo Wang,et al.  Dynamic Label Propagation for Semi-supervised Multi-class Multi-label Classification , 2013, ICCV.

[48]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.