Transformations for variational factor analysis to speed up learning

We propose simple transformation of the hidden states in variational Bayesian factor analysis models to speed up the learning procedure. The speed-up is achieved by using proper parameterization of the posterior approximation which allows joint optimization of its individual factors, thus the transformation is theor etically justified. We derive the transformation formulae f or variational Bayesian factor analysis and show experimentally that it can significantly improve the rate of convergence. The propose d transformation basically performs centering and whitening of the hidden factors taking into account the posterior uncertai nties. Similar transformations can be applied to other variational Bayesian factor analysis models as well.

[1]  Christopher M. Bishop.  Latent Variable Models , 1998, Learning in Graphical Models.

[2]  Andreas Ziehe,et al.  TDSEP { an e(cid:14)cient algorithm for blind separation using time structure , 1998 .

[3]  V. Smidl,et al.  Fast variational PCA for functional analysis of dynamic image sequences , 2003, 3rd International Symposium on Image and Signal Processing and Analysis, 2003. ISPA 2003. Proceedings of the.

[4]  Andrzej Cichocki,et al.  On-line Algorithm for Blind Signal Extraction of Arbitrarily Distributed, but Temporally Correlated Sources Using Second Order Statistics , 2000, Neural Processing Letters.

[5]  Jun S. Liu,et al.  Parameter Expansion for Data Augmentation , 1999 .

[6]  Yuan Qi,et al.  Parameter Expanded Variational Bayesian Methods , 2006, NIPS.

[7]  Michael E. Tipping,et al.  Probabilistic Principal Component Analysis , 1999 .

[8]  Frank-Michael Schleif,et al.  Advances in computational intelligence and learning , 2010 .

[9]  Katherine A. Heller,et al.  Bayesian Exponential Family PCA , 2008, NIPS.

[10]  Geoffrey E. Hinton,et al.  The EM algorithm for mixtures of factor analyzers , 1996 .

[11]  Michel Verleysen,et al.  Robust probabilistic projections , 2006, ICML.

[12]  Juha Karhunen,et al.  Accelerating Cyclic Update Algorithms for Parameter Estimation by Pattern Searches , 2003, Neural Processing Letters.

[13]  Geoffrey E. Hinton,et al.  Parameter estimation for linear dynamical systems , 1996 .

[14]  Michael I. Jordan.  Learning in Graphical Models , 1999, NATO ASI Series.

[15]  Charles M. Bishop.  Variational principal components , 1999 .

[16]  Antti Honkela,et al.  Unsupervised Variational Bayesian Learning of Nonlinear Models , 2004, NIPS.

[17]  Sanjoy Dasgupta,et al.  A Generalization of Principal Components Analysis to the Exponential Family , 2001, NIPS.

[18]  Antti Honkela,et al.  Bayesian Non-Linear Independent Component Analysis by Multi-Layer Perceptrons , 2000 .

[19]  Matthew J. Beal.  Variational algorithms for approximate Bayesian inference , 2003 .

[20]  Juha Karhunen,et al.  An Unsupervised Ensemble Learning Method for Nonlinear Dynamic State-Space Models , 2002, Neural Computation.

[21]  Alexander Basilevsky,et al.  Statistical Factor Analysis and Related Methods , 1994 .

[22]  D. Rubin,et al.  Parameter expansion to accelerate EM : The PX-EM algorithm , 1997 .

[23]  Xiao-Li Meng,et al.  The Art of Data Augmentation , 2001 .

[24]  M. Girolami,et al.  Advances in Independent Component Analysis , 2000, Perspectives in Neural Computing.

[25]  Hagai Attias,et al.  Independent Factor Analysis , 1999, Neural Computation.

[26]  Neil D. Lawrence,et al.  Probabilistic Non-linear Principal Component Analysis with Gaussian Process Latent Variable Models , 2005, J. Mach. Learn. Res..

[27]  Stan Lipovetsky,et al.  Latent Variable Models and Factor Analysis , 2001, Technometrics.

[28]  Geoffrey E. Hinton,et al.  A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants , 1998, Learning in Graphical Models.