Cumulative Restricted Boltzmann Machines for Ordinal Matrix Data Analysis

Ordinal data is omnipresent in almost all multiuser-generated feedback - questionnaires, preferences etc. This paper investigates modelling of ordinal data with Gaussian restricted Boltzmann machines (RBMs). In particular, we present the model architecture, learning and inference procedures for both vector-variate and matrix-variate ordinal data. We show that our model is able to capture latent opinion profile of citizens around the world, and is competitive against state-of-art collaborative filtering techniques on large-scale public datasets. The model thus has the potential to extend application of RBMs to diverse domains such as recommendation systems, product reviews and expert assessments.

[1]  Svetha Venkatesh,et al.  Ordinal Boltzmann Machines for Collaborative Filtering , 2009, UAI.

[2]  Geoffrey E. Hinton,et al.  Modeling pixel means and covariances using factorized third-order boltzmann machines , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[3]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[4]  Robert D. Mare,et al.  Social Background and School Continuation Decisions , 1980 .

[5]  Ruslan Salakhutdinov,et al.  Probabilistic Matrix Factorization , 2007, NIPS.

[6]  J A Anderson,et al.  The grouped continuous model for multivariate ordered categorical variables and covariate adjustment. , 1985, Biometrics.

[7]  S CardosoJaime,et al.  Learning to Classify Ordinal Data: The Data Replication Method , 2007 .

[8]  Wei Chu,et al.  Support Vector Ordinal Regression , 2007, Neural Computation.

[9]  Carla Rampichini,et al.  Alternative Specifications of Multivariate Multilevel Probit Ordinal Response Models , 2003 .

[10]  Svetha Venkatesh,et al.  A Sequential Decision Approach to Ordinal Preferences in Recommender Systems , 2012, AAAI.

[11]  P. McCullagh.  Regression Models for Ordinal Data , 1980 .

[12]  S. Chib,et al.  Analysis of multivariate probit models , 1998 .

[13]  Ivy Liu.  The Analysis of Ordered Categorical Data : An Overview and a Survey of Recent Developments , 2005 .

[14]  Yoshua Bengio,et al.  A Spike and Slab Restricted Boltzmann Machine , 2011, AISTATS.

[15]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[16]  Tijmen Tieleman,et al.  Training restricted Boltzmann machines using approximations to the likelihood gradient , 2008, ICML '08.

[17]  Peter V. Gehler,et al.  The rate adapting poisson model for information retrieval and object recognition , 2006, ICML.

[18]  David Haussler,et al.  Unsupervised learning of distributions on binary vectors using two layer networks , 1991, NIPS 1991.

[19]  Geoffrey E. Hinton,et al.  Semantic hashing , 2009, Int. J. Approx. Reason..

[20]  Geoffrey E. Hinton,et al.  Deep Boltzmann Machines , 2009, AISTATS.

[21]  Jaime S. Cardoso,et al.  Learning to Classify Ordinal Data: The Data Replication Method , 2007, J. Mach. Learn. Res..

[22]  Thore Graepel,et al.  Large Margin Rank Boundaries for Ordinal Regression , 2000 .

[23]  Nicolas Le Roux,et al.  Representational Power of Restricted Boltzmann Machines and Deep Belief Networks , 2008, Neural Computation.

[24]  Jean-Noël Bacro,et al.  A Hierarchical Bayesian Model for Spatial Prediction of Multivariate Non‐Gaussian Random Fields , 2011, Biometrics.

[25]  Geoffrey E. Hinton.  Training Products of Experts by Minimizing Contrastive Divergence , 2002, Neural Computation.

[26]  Paul Smolensky,et al.  Information processing in dynamical systems: foundations of harmony theory , 1986 .

[27]  Wei Chu,et al.  Gaussian Processes for Ordinal Regression , 2005, J. Mach. Learn. Res..

[28]  Quoc V. Le,et al.  Learning hierarchical invariant spatio-temporal features for action recognition with independent subspace analysis , 2011, CVPR 2011.

[29]  Geoffrey E. Hinton,et al.  Phone recognition using Restricted Boltzmann Machines , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[30]  Geoffrey E. Hinton,et al.  Restricted Boltzmann machines for collaborative filtering , 2007, ICML '07.

[31]  P. Müller,et al.  Nonparametric Bayesian Modeling for Multivariate Ordinal Data , 2005 .

[32]  Yoshua Bengio,et al.  Classification using discriminative restricted Boltzmann machines , 2008, ICML '08.

[33]  Geoffrey E. Hinton,et al.  Replicated Softmax: an Undirected Topic Model , 2009, NIPS.

[34]  Nando de Freitas,et al.  Inductive Principles for Restricted Boltzmann Machine Learning , 2010, AISTATS.

[35]  L. Younes.  Parametric Inference for imperfectly observed Gibbsian fields , 1989 .

[36]  Nathan Srebro,et al.  Fast maximum margin matrix factorization for collaborative prediction , 2005, ICML.

[37]  Svetha Venkatesh,et al.  Mixed-Variate Restricted Boltzmann Machines , 2014, ACML.

[38]  Yehuda Koren,et al.  OrdRec: an ordinal model for predicting personalized item rating distributions , 2011, RecSys '11.

[39]  Ivan Jeliazkov,et al.  FITTING AND COMPARISON OF MODELS FOR MULTIVARIATE ORDINAL OUTCOMES , 2008 .

[40]  János Podani,et al.  Multivariate exploratory analysis of ordinal data in ecology: Pitfalls, problems and solutions , 2005 .

[41]  C. Robert.  Simulation of truncated normal variables , 2009, 0907.4010.

[42]  Ole Winther,et al.  A hierarchical model for ordinal matrix factorization , 2012, Stat. Comput..

[43]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .