Beyond EM: A faster Bayesian linear regression algorithm without matrix inversions

Abstract The Bayesian linear regression is a useful tool for many scientific communities. This paper presents a novel algorithm for solving the Bayesian linear regression problem with Gaussian priors, which shares the same spirit as the gradient based methods. In addition, the standard scheme for this task, the Expectation Maximization (EM) algorithm, involves matrix inversions but our proposed algorithm is free of. Numerical experiments demonstrate that the proposed algorithm performs as well as the gradient based and EM algorithms in term of precision, but runs significantly faster than the gradient based and EM algorithms. Due to its matrix-inversion-free nature, the algorithm of this paper is a viable alternative to the competing methods available in the literature.

[1]  Aren Jansen,et al.  Fully unsupervised small-vocabulary speech recognition using a segmental Bayesian model , 2015, INTERSPEECH.

[2]  Boualem Boashash,et al.  Time-Frequency Signal Analysis and Processing: A Comprehensive Reference , 2015 .

[3]  Lennart Ljung,et al.  Implementation of algorithms for tuning parameters in regularized least squares problems in system identification , 2013, Autom..

[4]  Aggelos K. Katsaggelos,et al.  Variational Bayesian Blind Deconvolution Using a Total Variation Prior , 2009, IEEE Transactions on Image Processing.

[5]  Alessandro De Gloria,et al.  Time-Aware Multivariate Nearest Neighbor Regression Methods for Traffic Flow Prediction , 2015, IEEE Transactions on Intelligent Transportation Systems.

[6]  G. McLachlan,et al.  The EM algorithm and extensions , 1996 .

[7]  Tara N. Sainath,et al.  Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups , 2012, IEEE Signal Processing Magazine.

[8]  David Gesbert,et al.  A Coordinated Approach to Channel Estimation in Large-Scale Multiple-Antenna Systems , 2012, IEEE Journal on Selected Areas in Communications.

[9]  Cajo J. F. ter Braak,et al.  A Markov Chain Monte Carlo version of the genetic algorithm Differential Evolution: easy Bayesian computing for real parameter spaces , 2006, Stat. Comput..

[10]  Francesca P. Carli,et al.  Maximum Entropy Kernels for System Identification , 2014, IEEE Transactions on Automatic Control.

[11]  M. Giles.  Collected Matrix Derivative Results for Forward and Reverse Mode Algorithmic Differentiation , 2008 .

[12]  François Le Gall,et al.  Powers of tensors and fast matrix multiplication , 2014, ISSAC.

[13]  Guy-Bart Stan,et al.  A Sparse Bayesian Approach to the Identification of Nonlinear State-Space Systems , 2014, IEEE Transactions on Automatic Control.

[14]  Fei-Yue Wang,et al.  Traffic Flow Prediction With Big Data: A Deep Learning Approach , 2015, IEEE Transactions on Intelligent Transportation Systems.

[15]  Steven A. Tretter,et al.  Estimating the frequency of a noisy sinusoid by linear regression , 1985, IEEE Trans. Inf. Theory.

[16]  Gene H. Golub,et al.  Matrix computations , 1983 .

[17]  Jeffrey C. Lagarias,et al.  Convergence Properties of the Nelder-Mead Simplex Method in Low Dimensions , 1998, SIAM J. Optim..

[18]  David A. Clifton,et al.  Multitask Gaussian Processes for Multivariate Physiological Time-Series Analysis , 2015, IEEE Transactions on Biomedical Engineering.

[19]  Alberto Contreras-Cristán,et al.  A Bayesian Nonparametric Approach for Time Series Clustering , 2014 .

[20]  Brian Neelon,et al.  Bayesian Isotonic Regression and Trend Analysis , 2004, Biometrics.

[21]  Frédo Durand,et al.  Understanding and evaluating blind deconvolution algorithms , 2009, CVPR.