Unsupervised Variational Bayesian Learning of Nonlinear Models

In this paper we present a framework for using multi-layer perceptron (MLP) networks in nonlinear generative models trained by variational Bayesian learning. The nonlinearity is handled by linearizing it using a Gauss-Hermite quadrature at the hidden neurons. This yields an accurate approximation for cases of large posterior variance. The method can be used to derive nonlinear counterparts for linear algorithms such as factor analysis, independent component/factor analysis and state-space models. This is demonstrated with a nonlinear factor analysis experiment in which even 20 sources can be estimated from a real world speech data set.

[1]  Geoffrey E. Hinton,et al.  Keeping the neural networks simple by minimizing the description length of the weights , 1993, COLT '93.

[2]  S. Julier,et al.  A General Method for Approximating Nonlinear Transformations of Probability Distributions , 1996 .

[3]  David Barber,et al.  Ensemble Learning for Multi-Layer Networks , 1997, NIPS.

[4]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[5]  Hagai Attias,et al.  Independent Factor Analysis , 1999, Neural Computation.

[6]  Hagai Attias,et al.  A Variational Bayesian Framework for Graphical Models , 1999 .

[7]  Antti Honkela,et al.  Bayesian Non-Linear Independent Component Analysis by Multi-Layer Perceptrons , 2000 .

[8]  Francisco Curbera.  Delayed Curse of Dimension for Gaussian Integration , 2000, J. Complex..

[9]  Zoubin Ghahramani,et al.  Propagation Algorithms for Variational Bayesian Learning , 2000, NIPS.

[10]  Erkki Oja,et al.  Independent Component Analysis , 2001 .

[11]  Juha Karhunen,et al.  An Unsupervised Ensemble Learning Method for Nonlinear Dynamic State-Space Models , 2002, Neural Computation.

[12]  Stephen J. Roberts,et al.  Adaptive Classification by Variational Kalman Filtering , 2002, NIPS.

[13]  E. Oja,et al.  Nonlinear Blind Source Separation by Variational Bayesian Learning , 2003, IEICE Trans. Fundam. Electron. Commun. Comput. Sci..

[14]  A. Honkela.  Approximating nonlinear transformations of probability distributions for nonlinear independent component analysis , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).