Restricted Boltzmann Machines With Gaussian Visible Units Guided by Pairwise Constraints

Restricted Boltzmann machines (RBMs) and their variants are usually trained by contrastive divergence (CD) learning, but the training procedure is an unsupervised learning approach, without any guidances of the background knowledge. To enhance the expression ability of traditional RBMs, in this paper, we propose pairwise constraints (PCs) RBM with Gaussian visible units (pcGRBM) model, in which the learning procedure is guided by PCs and the process of encoding is conducted under these guidances. The PCs are encoded in hidden layer features of pcGRBM. Then, some pairwise hidden features of pcGRBM flock together and another part of them are separated by the guidances. In order to deal with real-valued data, the binary visible units are replaced by linear units with Gaussian noise in the pcGRBM model. In the learning process of pcGRBM, the PCs are iterated transitions between visible and hidden units during CD learning procedure. Then, the proposed model is inferred by approximative gradient descent method and the corresponding learning algorithm is designed. In order to compare the availability of pcGRBM and traditional RBMs with Gaussian visible units, the features of the pcGRBM and RBMs hidden layer are used as input “data” for ${K}$ -means, spectral clustering (SP) and affinity propagation (AP) algorithms, respectively. We also use tenfold cross-validation strategy to train and test pcGRBM model to obtain more meaningful results with PCs which are derived from incremental sampling procedures. A thorough experimental evaluation is performed with 12 image datasets of Microsoft Research Asia Multimedia. The experimental results show that the clustering performance of ${K}$ -means, SP, and AP algorithms based on pcGRBM model are significantly better than traditional RBMs. In addition, the pcGRBM model for clustering tasks shows better performance than some semi-supervised clustering algorithms.

[1]  Ming-Ai Li,et al.  A novel feature extraction method for scene recognition based on Centered Convolutional Restricted Boltzmann Machines , 2015, Neurocomputing.

[2]  David Haussler,et al.  Unsupervised learning of distributions on binary vectors using two layer networks , 1991, NIPS 1991.

[3]  Geoffrey E. Hinton,et al.  Learning Multilevel Distributed Representations for High-Dimensional Sequences , 2007, AISTATS.

[4]  Feng Liu,et al.  A joint-L2, 1-norm-constraint-based semi-supervised feature extraction for RNA-Seq data analysis , 2017, Neurocomputing.

[5]  Claire Cardie,et al.  Proceedings of the Eighteenth International Conference on Machine Learning, 2001, p. 577–584. Constrained K-means Clustering with Background Knowledge , 2022 .

[6]  Geoffrey E. Hinton,et al.  Replicated Softmax: an Undirected Topic Model , 2009, NIPS.

[7]  Xuelong Li,et al.  Unsupervised 3D Local Feature Learning by Circle Convolutional Restricted Boltzmann Machine , 2016, IEEE Transactions on Image Processing.

[8]  Thomas Hofmann,et al.  Greedy Layer-Wise Training of Deep Networks , 2007 .

[9]  Meng Wang,et al.  MSRA-MM 2.0: A Large-Scale Web Multimedia Dataset , 2009, 2009 IEEE International Conference on Data Mining Workshops.

[10]  Zhongzhi Shi,et al.  Incremental extreme learning machine based on deep feature embedded , 2016, Int. J. Mach. Learn. Cybern..

[11]  Daoqiang Zhang,et al.  Constraint Score: A new filter method for feature selection with pairwise constraints , 2008, Pattern Recognit..

[12]  Eric Eaton,et al.  Estimating 3D Trajectories from 2D Projections via Disjunctive Factored Four-Way Conditional Restricted Boltzmann Machines , 2017, Pattern Recognit..

[13]  Xing Zhao,et al.  Spectral–Spatial Classification of Hyperspectral Data Based on Deep Belief Network , 2015, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[14]  Qian Yu,et al.  Rényi Divergence Based Generalization for Learning of Classification Restricted Boltzmann Machines , 2014, 2014 IEEE International Conference on Data Mining Workshop.

[15]  Masato Okada,et al.  Dynamical analysis of contrastive divergence learning: Restricted Boltzmann machines with Gaussian visible units , 2016, Neural Networks.

[16]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[17]  Zhu Feng.  Semi-Supervised Dimensionality Reduction Algorithm of Tensor Image , 2009 .

[18]  Liang Chen,et al.  Semi-Supervised Learning Through Label Propagation on Geodesics , 2018, IEEE Transactions on Cybernetics.

[19]  Gang Chen,et al.  Deep Transductive Semi-supervised Maximum Margin Clustering , 2015, ArXiv.

[20]  G. X. Yu,et al.  Mixture graph based semi-supervised dimensionality reduction , 2010, Pattern Recognition and Image Analysis.

[21]  Chris H. Q. Ding,et al.  Orthogonal nonnegative matrix t-factorizations for clustering , 2006, KDD '06.

[22]  Geoffrey E. Hinton,et al.  Application of Deep Belief Networks for Natural Language Understanding , 2014, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[23]  Shingo Mabu,et al.  Time Series Prediction Using DBN and ARIMA , 2015, 2015 International Conference on Computer Application Technologies.

[24]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[25]  Geoffrey E. Hinton.  A Practical Guide to Training Restricted Boltzmann Machines , 2012, Neural Networks: Tricks of the Trade.

[26]  Ruslan Salakhutdinov,et al.  Learning Deep Generative Models , 2009 .

[27]  Laurenz Wiskott,et al.  Gaussian-binary restricted Boltzmann machines for modeling natural image statistics , 2014, PloS one.

[28]  S. P. Lloyd,et al.  Least squares quantization in PCM , 1982, IEEE Trans. Inf. Theory.

[29]  Qiang Ji,et al.  A generative restricted Boltzmann machine based method for high-dimensional motion data modeling , 2015, Comput. Vis. Image Underst..

[30]  Xiaolong Wang,et al.  Deep Belief Networks for Automatic Music Genre Classification , 2011, INTERSPEECH.

[31]  Yu Xiao,et al.  Semi-Supervised Clustering Based on Affinity Propagation Algorithm: Semi-Supervised Clustering Based on Affinity Propagation Algorithm , 2009 .

[32]  Kenji Doya,et al.  Expected energy-based restricted Boltzmann machine for classification , 2015, Neural Networks.

[33]  Sotirios A. Tsaftaris,et al.  Theta-RBM: Unfactored Gated Restricted Boltzmann Machine for Rotation-Invariant Representations , 2016, ArXiv.

[34]  Tetsuya Takiguchi,et al.  Non-Parallel Training in Voice Conversion Using an Adaptive Restricted Boltzmann Machine , 2016, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[35]  Lei Nie,et al.  Periocular Recognition Using Unsupervised Convolutional RBM Feature Learning , 2014, 2014 22nd International Conference on Pattern Recognition.

[36]  Lei Deng,et al.  Temperature based Restricted Boltzmann Machines , 2016, Scientific Reports.

[37]  C. L. Philip Chen,et al.  Fuzzy Restricted Boltzmann Machine for the Enhancement of Deep Learning , 2015, IEEE Transactions on Fuzzy Systems.

[38]  Qinghua Zheng,et al.  An Adaptive Semisupervised Feature Analysis for Video Semantic Recognition , 2018, IEEE Transactions on Cybernetics.

[39]  Geoffrey E. Hinton,et al.  The Recurrent Temporal Restricted Boltzmann Machine , 2008, NIPS.

[40]  Jakub M. Tomczak.  Learning Informative Features from Restricted Boltzmann Machines , 2015, Neural Processing Letters.

[41]  Honglak Lee,et al.  Sparse deep belief net model for visual area V2 , 2007, NIPS.

[42]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[43]  Yee Whye Teh,et al.  Rate-coded Restricted Boltzmann Machines for Face Recognition , 2000, NIPS.

[44]  A. Krizhevsky.  Convolutional Deep Belief Networks on CIFAR-10 , 2010 .

[45]  Yoshua Bengio,et al.  The Spike-and-Slab RBM and Extensions to Discrete and Sparse Data Distributions , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[46]  Tapani Raiko,et al.  Gaussian-Bernoulli deep Boltzmann machine , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[47]  Jürgen Schmidhuber,et al.  Multi-column deep neural networks for image classification , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[48]  Geoffrey E. Hinton,et al.  Modeling image patches with a directed hierarchy of Markov random fields , 2007, NIPS.

[49]  Dejing Dou,et al.  Social restricted Boltzmann Machine: Human behavior prediction in health social networks , 2015, 2015 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM).

[50]  Pascal Vincent,et al.  Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion , 2010, J. Mach. Learn. Res..

[51]  Michael I. Jordan,et al.  On Spectral Clustering: Analysis and an algorithm , 2001, NIPS.

[52]  Kunikazu Kobayashi,et al.  Time Series Forecasting Using Restricted Boltzmann Machine , 2012, ICIC.

[53]  Geoffrey E. Hinton,et al.  Learning and relearning in Boltzmann machines , 1986 .

[54]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[55]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[56]  Haizhou Li,et al.  Real-Time Keypoint Recognition Using Restricted Boltzmann Machine , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[57]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[58]  Geoffrey E. Hinton,et al.  Conditional Restricted Boltzmann Machines for Structured Output Prediction , 2011, UAI.

[59]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[60]  Kunikazu Kobayashi,et al.  Time series forecasting using a deep belief network with restricted Boltzmann machines , 2014, Neurocomputing.

[61]  Geoffrey E. Hinton,et al.  Speech recognition with deep recurrent neural networks , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[62]  Enrico Zio,et al.  Fuzzy Classification With Restricted Boltzman Machines and Echo-State Networks for Predicting Potential Railway Door System Failures , 2015, IEEE Transactions on Reliability.

[63]  Delbert Dueck,et al.  Clustering by Passing Messages Between Data Points , 2007, Science.

[64]  Geoffrey E. Hinton,et al.  An Efficient Learning Procedure for Deep Boltzmann Machines , 2012, Neural Computation.

[65]  Daoqiang Zhang,et al.  Constraint Projections for Ensemble Learning , 2008, AAAI.

[66]  Tapani Raiko,et al.  Improved Learning of Gaussian-Bernoulli Restricted Boltzmann Machines , 2011, ICANN.

[67]  Daoqiang Zhang,et al.  Semisupervised Dimensionality Reduction With Pairwise Constraints for Hyperspectral Image Classification , 2011, IEEE Geoscience and Remote Sensing Letters.

[68]  Yadong Mu,et al.  Supervised deep learning with auxiliary networks , 2014, KDD.

[69]  Tieniu Tan,et al.  Learning Relevance Restricted Boltzmann Machine for Unstructured Group Activity and Event Understanding , 2016, International Journal of Computer Vision.

[70]  Francisco Herrera,et al.  Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power , 2010, Inf. Sci..

[71]  Geoffrey E. Hinton.  Training Products of Experts by Minimizing Contrastive Divergence , 2002, Neural Computation.

[72]  Geoffrey E. Hinton,et al.  OPTIMAL PERCEPTUAL INFERENCE , 1983 .

[73]  Matthias Hein,et al.  Constrained 1-Spectral Clustering , 2012, AISTATS.

[74]  Miguel Á. Carreira-Perpiñán,et al.  On Contrastive Divergence Learning , 2005, AISTATS.

[75]  Matthieu Cord,et al.  Learning Deep Hierarchical Visual Feature Coding , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[76]  Geoffrey E. Hinton,et al.  Two Distributed-State Models For Generating High-Dimensional Time Series , 2011, J. Mach. Learn. Res..

[77]  Pascal Vincent,et al.  Representation Learning: A Review and New Perspectives , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[78]  Nitish Srivastava,et al.  Modeling Documents with Deep Boltzmann Machines , 2013, UAI.

[79]  Tetsuya Takiguchi,et al.  Voice Conversion Using RNN Pre-Trained by Recurrent Temporal Restricted Boltzmann Machines , 2015, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[80]  VincentPascal,et al.  Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion , 2010 .

[81]  Jiawei Han,et al.  Document clustering using locality preserving indexing , 2005, IEEE Transactions on Knowledge and Data Engineering.