Mixed-Variate Restricted Boltzmann Machines

Modern datasets are becoming heterogeneous. To this end, we present in this paper MixedVariate Restricted Boltzmann Machines for simultaneously modelling variables of multiple types and modalities, including binary and continuous responses, categorical options, multicategorical choices, ordinal assessment and category-ranked preferences. Dependency among variables is modeled using latent binary variables, each of which can be interpreted as a particular hidden aspect of the data. The proposed model, similar to the standard RBMs, allows fast evaluation of the posterior for the latent variables. Hence, it is naturally suitable for many common tasks including, but not limited to, (a) as a pre-processing step to convert complex input data into a more convenient vectorial representation through the latent posteriors, thereby oering a dimensionality reduction capacity, (b) as a classier supporting binary, multiclass, multilabel, and label-ranking outputs, or a regression tool for continuous outputs and (c) as a data completion tool for multimodal and heterogeneous data. We evaluate the proposed model on a large-scale dataset using the world opinion survey results on three tasks: feature extraction and visualization, data completion and prediction.

[1]  J. Anderson.  Regression and Ordered Categorical Variables , 1984 .

[2]  Yoshua Bengio,et al.  Classification using discriminative restricted Boltzmann machines , 2008, ICML '08.

[3]  Peter V. Gehler,et al.  The rate adapting poisson model for information retrieval and object recognition , 2006, ICML.

[4]  R. Davidson.  On Extending the Bradley-Terry Model to Accommodate Ties in Paired Comparison Experiments , 1970 .

[5]  Eyke Hüllermeier,et al.  Label ranking by learning pairwise preferences , 2008, Artif. Intell..

[6]  L. Ryan,et al.  Latent Variable Models for Mixed Discrete and Continuous Outcomes , 1997 .

[7]  Jieping Ye,et al.  Extracting shared subspace for multi-label classification , 2008, KDD.

[8]  Geoffrey E. Hinton,et al.  Deep Boltzmann Machines , 2009, AISTATS.

[9]  Geoffrey E. Hinton,et al.  Learning and relearning in Boltzmann machines , 1986 .

[10]  D. Dunson,et al.  Bayesian latent variable models for clustered mixed outcomes , 2000 .

[11]  M. Muresan.  A concrete approach to classical analysis , 2009 .

[12]  David Haussler,et al.  Unsupervised learning of distributions on binary vectors using two layer networks , 1991, NIPS 1991.

[13]  Geoffrey E. Hinton.  Training Products of Experts by Minimizing Contrastive Divergence , 2002, Neural Computation.

[14]  C. McCulloch.  Joint modelling of mixed outcome types using latent variables , 2008, Statistical methods in medical research.

[15]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[16]  Svetha Venkatesh,et al.  Ordinal Boltzmann Machines for Collaborative Filtering , 2009, UAI.

[17]  Nicolas Le Roux,et al.  Learning a Generative Model of Images by Factoring Appearance and Shape , 2011, Neural Computation.

[18]  Geoffrey E. Hinton,et al.  Modeling pixel means and covariances using factorized third-order boltzmann machines , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[19]  Hans-Peter Kriegel,et al.  Collaborative ordinal regression , 2006, ICML.

[20]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[21]  P. McCullagh.  Regression Models for Ordinal Data , 1980 .

[22]  S Y Lee,et al.  Latent variable models with mixed continuous and polytomous data , 2001, Biometrics.

[23]  Yoram Singer,et al.  Log-Linear Models for Label Ranking , 2003, NIPS.

[24]  Geoffrey E. Hinton,et al.  Restricted Boltzmann machines for collaborative filtering , 2007, ICML '07.

[25]  Rong Yan,et al.  Mining Associated Text and Images with Dual-Wing Harmoniums , 2005, UAI.

[26]  Geoffrey E. Hinton,et al.  Replicated Softmax: an Undirected Topic Model , 2009, NIPS.

[27]  Amnon Shashua,et al.  Ranking with Large Margin Principle: Two Approaches , 2002, NIPS.

[28]  Grigorios Tsoumakas,et al.  Multi-Label Classification: An Overview , 2007, Int. J. Data Warehous. Min..

[29]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.