An Unsupervised Ensemble Learning Method for Nonlinear Dynamic State-Space Models

A Bayesian ensemble learning method is introduced for unsupervised extraction of dynamic processes from noisy data. The data are assumed to be generated by an unknown nonlinear mapping from unknown factors. The dynamics of the factors are modeled using a nonlinear state-space model. The nonlinear mappings in the model are represented using multilayer perceptron networks. The proposed method is computationally demanding, but it allows the use of higher-dimensional nonlinear latent variable models than other existing approaches. Experiments with chaotic data show that the new method is able to blindly estimate the factors and the dynamic process that generated the data. It clearly outperforms currently available nonlinear prediction techniques in this very difficult test problem.

[1]  E. Lorenz.  Deterministic nonperiodic flow , 1963 .

[2]  Harold W. Sorenson,et al.  Parameter estimation: Principles and problems , 1980 .

[3]  F. Takens.  Detecting strange attractors in turbulence , 1981 .

[4]  Martin Casdagli,et al.  Nonlinear prediction of chaotic time series , 1989 .

[5]  David J. C. MacKay,et al.  A Practical Bayesian Framework for Backpropagation Networks , 1992, Neural Computation.

[6]  Geoffrey E. Hinton,et al.  Keeping the neural networks simple by minimizing the description length of the weights , 1993, COLT '93.

[7]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[8]  David B. Dunson,et al.  Bayesian Data Analysis , 2010 .

[9]  Geoffrey E. Hinton,et al.  Bayesian Learning for Neural Networks , 1995 .

[10]  David J. C. MacKay,et al.  Developments in Probabilistic Modelling with Neural Networks - Ensemble Learning , 1995, SNN Symposium on Neural Networks.

[11]  Ole Winther,et al.  Bayesian Mean Field Algorithms for Neural Networks and Gaussian Processes , 1998 .

[12]  Simon Haykin,et al.  Making sense of a complex world , 1998 .

[13]  Hagai Attias.  Learning a Hierarchical Belief Network of Independent Factor Analyzers , 1998, NIPS.

[14]  Zoubin Ghahramani,et al.  Learning Nonlinear Dynamical Systems Using an EM Algorithm , 1998, NIPS.

[15]  Michael I. Jordan.  Learning in Graphical Models , 1999, NATO ASI Series.

[16]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[17]  Charles M. Bishop,et al.  Ensemble learning in Bayesian neural networks , 1998 .

[18]  Volker Tresp,et al.  Fisher Scoring and a Mixture of Modes Approach for Approximate Inference and Learning in Nonlinear State Space Models , 1998, NIPS.

[19]  S. Haykin,et al.  Making sense of a complex world [chaotic events modeling] , 1998, IEEE Signal Process. Mag..

[20]  Michael I. Jordan,et al.  An Introduction to Variational Methods for Graphical Models , 1999, Machine-mediated learning.

[21]  S. Bernstein,et al.  Introduction , 1999, Brain and Language.

[22]  Hagai Attias,et al.  Independent Factor Analysis , 1999, Neural Computation.

[23]  최승진.  Nonlinear dynamic independent component analysis using state-space and neural network models , 1999 .

[24]  Harri Lappalainen,et al.  Ensemble learning for independent component analysis , 1999 .

[25]  Hagai Attias,et al.  Independent Factor Analysis with Temporally Structured Sources , 1999, NIPS.

[26]  Jose C. Principe,et al.  Neural and Adaptive Systems: Fundamentals through Simulations with CD-ROM , 1999 .

[27]  Harri Valpola,et al.  Bayesian Ensemble Learning for Nonlinear Factor Analysis , 2000 .

[28]  Stephen J. Roberts,et al.  An ensemble learning approach to independent component analysis , 2000, Neural Networks for Signal Processing X. Proceedings of the 2000 IEEE Signal Processing Society Workshop (Cat. No.00TH8501).

[29]  Antti Honkela,et al.  Bayesian Non-Linear Independent Component Analysis by Multi-Layer Perceptrons , 2000 .

[30]  Nikunj C. Oza,et al.  Online Ensemble Learning , 2000, AAAI/IAAI.

[31]  M. Girolami,et al.  Advances in Independent Component Analysis , 2000, Perspectives in Neural Computing.

[32]  Harri Valpola.  Nonlinear independent component analysis using ensemble learning: Theory , 2000 .

[33]  David J. C. MacKay,et al.  Ensemble Learning for Blind Image Separation and Deconvolution , 2000 .

[34]  Juha Karhunen,et al.  Nonlinear Independent Component Analysis Using Ensemble Learning: Experiments and Discussion , 2000 .

[35]  Zoubin Ghahramani,et al.  Propagation Algorithms for Variational Bayesian Learning , 2000, NIPS.

[36]  H. Attias.  Independent Component Analysis: ICA, graphical models and variational methods , 2001 .

[37]  Juha Karhunen,et al.  An Ensemble Learning Approach to Nonlinear Independent Component Analysis , 2001 .

[38]  Erkki Oja,et al.  DYNAMICAL FACTOR ANALYSIS OF RHYTHMIC MAGNETOENCEPHALOGRAPHIC ACTIVITY , 2001 .

[39]  Zoubin Ghahramani,et al.  An Introduction to Hidden Markov Models and Bayesian Networks , 2001, Int. J. Pattern Recognit. Artif. Intell..

[40]  J. Karhunen,et al.  Building Blocks for Hierarchical Latent Variable Models , 2001 .

[41]  Terrence J. Sejnowski,et al.  Variational Learning for Switching State-Space Models , 2001 .

[42]  E. Wan,et al.  5 DUAL EXTENDED KALMAN FILTER METHODS , 2001 .

[43]  Jouko Lampinen,et al.  Bayesian approach for neural networks--review and case studies , 2001, Neural Networks.

[44]  J. W. Miskin,et al.  Independent Component Analysis: Ensemble Learning for blind source separation , 2001 .

[45]  V. Kvasnicka,et al.  Neural and Adaptive Systems: Fundamentals Through Simulations , 2001, IEEE Trans. Neural Networks.

[46]  Erkki Oja,et al.  Independent Component Analysis , 2001 .

[47]  Rudolph van der Merwe,et al.  The Unscented Kalman Filter , 2002 .

[48]  Eric A. Wan,et al.  Dual Extended Kalman Filter Methods , 2002 .

[49]  Erkki Oja,et al.  DETECTING PROCESS STATE CHANGES BY NONLINEAR BLIND SOURCE SEPARATION , 2003 .

[50]  William S. Rayens,et al.  Independent Component Analysis: Principles and Practice , 2003, Technometrics.

[51]  Christian Jutten,et al.  Three easy ways for separating nonlinear mixtures? , 2004, Signal Process..

[52]  Juha Karhunen,et al.  Hierarchical models of variance sources , 2004, Signal Process..

[53]  Christian P. Robert,et al.  Monte Carlo Statistical Methods , 2005, Springer Texts in Statistics.