Hierarchical models of variance sources

In many models, variances are assumed to be constant although this assumption is often unrealistic in practice. Joint modelling of means and variances is difficult in many learning approaches, because it can lead into infinite probability densities. We show that a Bayesian variational technique which is sensitive to probability mass instead of density is able to jointly model both variances and means. We consider a model structure where a Gaussian variable, called variance node, controls the variance of another Gaussian variable. Variance nodes make it possible to build hierarchical models for both variances and means. We report experiments with artificial data which demonstrate the ability of the learning algorithm to find variance sources explaining and characterizing well the variances in the multidimensional data. Experiments with biomedical MEG data show that variance sources are present in real-world signals.

[1]  E. Oja,et al.  Independent Component Analysis , 2013 .

[2]  Aapo Hyvärinen,et al.  Blind separation of sources that have spatiotemporal variance dependencies , 2004, Signal Process..

[3]  Geoffrey E. Hinton,et al.  Keeping the neural networks simple by minimizing the description length of the weights , 1993, COLT '93.

[4]  Charles M. Bishop,et al.  Ensemble learning in Bayesian neural networks , 1998 .

[5]  N. Shephard,et al.  Stochastic Volatility: Likelihood Inference And Comparison With Arch Models , 1996 .

[6]  A. Harvey,et al.  5 Stochastic volatility , 1996 .

[7]  Lucas C. Parra,et al.  Higher-Order Statistical Properties Arising from the Non-Stationarity of Natural Signals , 2000, NIPS.

[8]  David Barber,et al.  Ensemble Learning for Multi-Layer Networks , 1997, NIPS.

[9]  Samuel Kaski,et al.  Self-Organized Formation of Various Invariant-Feature Filters in the Adaptive-Subspace SOM , 1997, Neural Computation.

[10]  J. Karhunen,et al.  Nonlinear Independent Factor Analysis by Hierarchical Models , 2003 .

[11]  Erkki Oja,et al.  Independent component approach to the analysis of EEG and MEG recordings , 2000, IEEE Transactions on Biomedical Engineering.

[12]  M. Girolami,et al.  Advances in Independent Component Analysis , 2000, Perspectives in Neural Computing.

[13]  J. Karhunen,et al.  Building Blocks for Hierarchical Latent Variable Models , 2001 .

[14]  Hagai Attias,et al.  Independent Factor Analysis , 1999, Neural Computation.

[15]  Jean-François Cardoso,et al.  Multidimensional independent component analysis , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[16]  L. Lathauwer,et al.  Fetal electrocardiogram extraction by source subspace separation , 1995 .

[17]  Nikunj C. Oza,et al.  Online Ensemble Learning , 2000, AAAI/IAAI.

[18]  Terrence J. Sejnowski,et al.  Variational Learning of Clusters of Undercomplete Nonsymmetric Independent Components , 2003, J. Mach. Learn. Res..

[19]  Juha Karhunen,et al.  Accelerating Cyclic Update Algorithms for Parameter Estimation by Pattern Searches , 2003, Neural Processing Letters.

[20]  T. Bollerslev,et al.  Generalized autoregressive conditional heteroskedasticity , 1986 .

[21]  Mark A. Girolami,et al.  A Variational Method for Learning Sparse and Overcomplete Representations , 2001, Neural Computation.

[22]  Calyampudi Radhakrishna Rao,et al.  Statistical methods in finance , 1996 .

[23]  Geoffrey E. Hinton,et al.  Variational Learning for Switching State-Space Models , 2000, Neural Computation.

[24]  Stephen J. Roberts,et al.  An ensemble learning approach to independent component analysis , 2000, Neural Networks for Signal Processing X. Proceedings of the 2000 IEEE Signal Processing Society Workshop (Cat. No.00TH8501).

[25]  Michael I. Jordan.  Learning in Graphical Models , 1999, NATO ASI Series.

[26]  Alexander Ilin,et al.  On the Effect of the Form of the Posterior Approximation in Variational Learning of ICA Models , 2005, Neural Processing Letters.

[27]  David J. C. MacKay,et al.  Ensemble Learning for Blind Image Separation and Deconvolution , 2000 .

[28]  Aapo Hyvärinen,et al.  Emergence of Phase- and Shift-Invariant Features by Decomposition of Natural Images into Independent Feature Subspaces , 2000, Neural Computation.

[29]  Juha Karhunen,et al.  An Unsupervised Ensemble Learning Method for Nonlinear Dynamic State-Space Models , 2002, Neural Computation.

[30]  Christopher M. Bishop,et al.  Neural networks and machine learning , 1998 .