Variational bounds for mixed-data factor analysis

We propose a new variational EM algorithm for fitting factor analysis models with mixed continuous and categorical observations. The algorithm is based on a simple quadratic bound to the log-sum-exp function. In the special case of fully observed binary data, the bound we propose is significantly faster than previous variational methods. We show that EM is significantly more robust in the presence of missing data compared to treating the latent factors as parameters, which is the approach used by exponential family PCA and other related matrix-factorization methods. A further benefit of the variational approach is that it can easily be extended to the case of mixtures of factor analyzers, as we show. We present results on synthetic and real data sets demonstrating several desirable properties of our proposed method.

[1]  D. Böhning.  Multinomial logistic regression algorithm , 1992 .

[2]  Geoffrey E. Hinton,et al.  The EM algorithm for mixtures of factor analyzers , 1996 .

[3]  Michael I. Jordan,et al.  A Variational Approach to Bayesian Logistic Regression Models and their Extensions , 1997, AISTATS.

[4]  Sam T. Roweis,et al.  EM Algorithms for PCA and SPCA , 1997, NIPS.

[5]  Michael E. Tipping.  Probabilistic Visualisation of High-Dimensional Binary Data , 1998, NIPS.

[6]  Michael E. Tipping,et al.  Probabilistic Principal Component Analysis , 1999 .

[7]  M. Wedel,et al.  Factor analysis with (mixed) observed and latent variables in the exponential family , 2001 .

[8]  Sanjoy Dasgupta,et al.  A Generalization of Principal Components Analysis to the Exponential Family , 2001, NIPS.

[9]  Antti Honkela,et al.  A state-space method for language modeling , 2003, 2003 IEEE Workshop on Automatic Speech Recognition and Understanding (IEEE Cat. No.03EX721).

[10]  David A. Forsyth,et al.  Matching Words and Pictures , 2003, J. Mach. Learn. Res..

[11]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[12]  Aleks Jakulin,et al.  Applying Discrete PCA in Data Analysis , 2004, UAI.

[13]  Volker Tresp,et al.  Heterogenous Data Fusion via a Probabilistic Latent-Variable Model , 2004, ARCS.

[14]  John D. Lafferty,et al.  Correlated Topic Models , 2005, NIPS.

[15]  Guillaume Bouchard.  Efficient Bounds for the Softmax Function and Applications to Approximate Inference in Hybrid models , 2008 .

[16]  Eric P. Xing,et al.  Seeking The Truly Correlated Topic Posterior - on tight approximate inference of logistic-normal admixture model , 2007, AISTATS.

[17]  Amr Ahmed,et al.  On Tight Approximate Inference of the Logistic-Normal Topic Admixture Model , 2007 .

[18]  David B Dunson,et al.  Bayesian methods for latent trait modelling of longitudinal data , 2007, Statistical methods in medical research.

[19]  C. Carvalho,et al.  Sparse Factor-Analytic Probit Models , 2008 .

[20]  Max Welling,et al.  Deterministic Latent Variable Models and Their Pitfalls , 2008, SDM.

[21]  Katherine A. Heller,et al.  Bayesian Exponential Family PCA , 2008, NIPS.

[22]  Ben Calderhead,et al.  Riemannian Manifold Hamiltonian Monte Carlo , 2009, 0907.1100.

[23]  H. Rue,et al.  Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations , 2009 .

[24]  R. Kass,et al.  Approximate Methods for State-Space Models , 2010, Journal of the American Statistical Association.