On Bayesian principal component analysis

A complete Bayesian framework for principal component analysis (PCA) is proposed. Previous model-based approaches to PCA were often based upon a factor analysis model with isotropic Gaussian noise. In contrast to PCA, these approaches do not impose orthogonality constraints. A new model with orthogonality restrictions is proposed. Its approximate Bayesian solution using the variational approximation and results from directional statistics is developed. The Bayesian solution provides two notable results in relation to PCA. The first is uncertainty bounds on principal components (PCs), and the second is an explicit distribution on the number of relevant PCs. The posterior distribution of the PCs is found to be of the von-Mises-Fisher type. This distribution and its associated hypergeometric function, F10, are studied. Numerical reductions are revealed, leading to a stable and efficient orthogonal variational PCA (OVPCA) algorithm. OVPCA provides the required inferences. Its performance is illustrated in simulation, and for a sequence of medical scintigraphic images.

[1]  Anja Vogler,et al.  An Introduction to Multivariate Statistical Analysis , 2004 .

[2]  Irene A. Stegun,et al.  Handbook of Mathematical Functions. , 1966 .

[3]  R. Baierlein.  Probability Theory: The Logic of Science , 2004 .

[4]  A. James.  Distributions of Matrix Variates and Latent Roots Derived from Normal Samples , 1964 .

[5]  Ronald W. Butler,et al.  Laplace approximation for Bessel functions of matrix argument , 2003 .

[6]  I. Jolliffe.  Principal Component Analysis , 2002 .

[7]  H. Jeffreys,et al.  Theory of probability , 1896 .

[8]  Charles M. Bishop.  Variational principal components , 1999 .

[9]  Matthew J. Beal,et al.  The variational Bayesian EM algorithm for incomplete data: with application to scoring graphical model structures , 2003 .

[10]  R. A. Leibler,et al.  On Information and Sufficiency , 1951 .

[11]  A. Sampson,et al.  Contributions to Probability and Statistics , 1989 .

[12]  S. Kay.  Fundamentals of statistical signal processing: estimation theory , 1993 .

[13]  H. Hotelling.  Analysis of a complex of statistical variables into principal components. , 1933 .

[14]  Zoubin Ghahramani,et al.  Variational Inference for Bayesian Mixtures of Factor Analysers , 1999, NIPS.

[15]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[16]  Dorothy T. Thayer,et al.  EM algorithms for ML factor analysis , 1982 .

[17]  I. Buvat,et al.  A statistical model for the determination of the optimal metric in factor analysis of medical image sequences (FAMIS) , 1993 .

[18]  I Buvat,et al.  Statistical distribution of factors and factor images in factor analysis of medical image sequences. , 1998, Physics in medicine and biology.

[19]  Gene H. Golub,et al.  Matrix computations , 1983 .

[20]  K. Mardia,et al.  The von Mises–Fisher Matrix Distribution in Orientation Statistics , 1977 .

[21]  Michael E. Tipping,et al.  Probabilistic Principal Component Analysis , 1999 .

[22]  Christopher M. Bishop,et al.  Mixtures of Probabilistic Principal Component Analyzers , 1999, Neural Computation.

[23]  Harri Lappalainen,et al.  Ensemble learning for independent component analysis , 1999 .

[24]  Masa-aki Sato,et al.  Online Model Selection Based on the Variational Bayes , 2001, Neural Computation.

[25]  Karl Pearson F.R.S..  LIII. On lines and planes of closest fit to systems of points in space , 1901 .

[26]  Yen-Wei Chen,et al.  Ensemble learning for independent component analysis , 2006, Pattern Recognit..

[27]  S. J. Press,et al.  Bayesian Inference in Factor Analysis , 1989 .

[28]  Anthony Quinn.  Regularized Signal Identification Using Bayesian Techniques , 1998 .

[29]  Hagai Attias,et al.  A Variational Bayesian Framework for Graphical Models , 1999 .