Transformations in variational Bayesian factor analysis to speed up learning

We propose simple transformation of the hidden states in variational Bayesian factor analysis models to speed up the learning procedure. The speed-up is achieved by using proper parameterization of the posterior approximation which allows joint optimization of its individual factors, thus the transformation is theoretically justified. We derive the transformation formulae for variational Bayesian factor analysis and show experimentally that it can significantly improve the rate of convergence. The proposed transformation basically performs centering and whitening of the hidden factors taking into account the posterior uncertainties. Similar transformations can be applied to other variational Bayesian factor analysis models as well.

[1]  Michel Verleysen,et al.  Robust probabilistic projections , 2006, ICML.

[2]  Geoffrey E. Hinton,et al.  Parameter estimation for linear dynamical systems , 1996 .

[3]  Xiao-Li Meng,et al.  The Art of Data Augmentation , 2001 .

[4]  Charles M. Bishop.  Variational principal components , 1999 .

[5]  Antti Honkela,et al.  Unsupervised Variational Bayesian Learning of Nonlinear Models , 2004, NIPS.

[6]  Katherine A. Heller,et al.  Bayesian Exponential Family PCA , 2008, NIPS.

[7]  Juha Karhunen,et al.  Bayesian Robust PCA of Incomplete Data , 2009, Neural Processing Letters.

[8]  Christopher M. Bishop.  Latent Variable Models , 1998, Learning in Graphical Models.

[9]  Nasser M. Nasrabadi,et al.  Pattern Recognition and Machine Learning , 2006, Technometrics.

[10]  Matthew J. Beal,et al.  The variational Bayesian EM algorithm for incomplete data: with application to scoring graphical model structures , 2003 .

[11]  Matthew J. Beal.  Variational algorithms for approximate Bayesian inference , 2003 .

[12]  Geoffrey E. Hinton,et al.  The EM algorithm for mixtures of factor analyzers , 1996 .

[13]  E. B. Andersen,et al.  Information Science and Statistics , 1986 .

[14]  V. Smidl,et al.  Fast variational PCA for functional analysis of dynamic image sequences , 2003, 3rd International Symposium on Image and Signal Processing and Analysis, 2003. ISPA 2003. Proceedings of the.

[15]  Jong-Hoon Ahn,et al.  A Constrained EM Algorithm for Principal Component Analysis , 2003, Neural Computation.

[16]  Jun S. Liu,et al.  Parameter Expansion for Data Augmentation , 1999 .

[17]  Yuan Qi,et al.  Parameter Expanded Variational Bayesian Methods , 2006, NIPS.

[18]  Alexander Basilevsky,et al.  Statistical Factor Analysis and Related Methods , 1994 .

[19]  Tapani Raiko,et al.  Transformations for variational factor analysis to speed up learning , 2009, ESANN.

[20]  D. Bartholomew.  Latent Variable Models And Factor Analysis , 1987 .

[21]  M. Girolami,et al.  Advances in Independent Component Analysis , 2000, Perspectives in Neural Computing.

[22]  Ruslan Salakhutdinov,et al.  Bayesian probabilistic matrix factorization using Markov chain Monte Carlo , 2008, ICML '08.

[23]  Michael E. Tipping,et al.  Probabilistic Principal Component Analysis , 1999 .

[24]  Hagai Attias,et al.  Independent Factor Analysis , 1999, Neural Computation.

[25]  Christopher M. Bishop,et al.  Learning in Graphical Models , 1999 .

[26]  Antti Honkela,et al.  Bayesian Non-Linear Independent Component Analysis by Multi-Layer Perceptrons , 2000 .

[27]  Juha Karhunen,et al.  An Unsupervised Ensemble Learning Method for Nonlinear Dynamic State-Space Models , 2002, Neural Computation.

[28]  Neil D. Lawrence,et al.  Probabilistic Non-linear Principal Component Analysis with Gaussian Process Latent Variable Models , 2005, J. Mach. Learn. Res..

[29]  Juha Karhunen,et al.  Accelerating Cyclic Update Algorithms for Parameter Estimation by Pattern Searches , 2003, Neural Processing Letters.

[30]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[31]  Geoffrey E. Hinton,et al.  A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants , 1998, Learning in Graphical Models.

[32]  Hagai Attias,et al.  A Variational Bayesian Framework for Graphical Models , 1999 .