Bayesian Methods for Low-Rank Matrix Estimation: Short Survey and Theoretical Study

The problem of low-rank matrix estimation recently received a lot of attention due to challenging applications. A lot of work has been done on rank-penalized methods and convex relaxation, both on the theoretical and applied sides. However, only a few papers considered Bayesian estimation. In this paper, we review the different type of priors considered on matrices to favour low-rank. We also prove that the obtained Bayesian estimators, under suitable assumptions, enjoys the same optimality properties as the ones based on penalization.

[1]  G. Reinsel,et al.  Multivariate Reduced-Rank Regression: Theory and Applications , 1998 .

[2]  Emmanuel J. Candès,et al.  Matrix Completion With Noise , 2009, Proceedings of the IEEE.

[3]  Emmanuel J. Candès,et al.  The Power of Convex Relaxation: Near-Optimal Matrix Completion , 2009, IEEE Transactions on Information Theory.

[4]  Arnak S. Dalalyan,et al.  Sparse Regression Learning by Aggregation and Langevin Monte-Carlo , 2009, COLT.

[5]  T. W. Anderson.  Estimating Linear Restrictions on Regression Coefficients for Multivariate Normal Distributions , 1951 .

[6]  A. Tsybakov,et al.  Estimation of high-dimensional low-rank matrices , 2009, 0912.5338.

[7]  A. Izenman.  Reduced-rank regression for the multivariate linear model , 1975 .

[8]  Neil D. Lawrence,et al.  Non-linear matrix factorization with Gaussian processes , 2009, ICML '09.

[9]  Emmanuel J. Candès,et al.  Exact Matrix Completion via Convex Optimization , 2009, Found. Comput. Math..

[10]  V. Koltchinskii,et al.  Oracle inequalities in empirical risk minimization and sparse recovery problems , 2011 .

[11]  Richard Paap,et al.  Priors, posteriors and Bayes factors for a Bayesian analysis of cointegration , 1998 .

[12]  M. Wegkamp,et al.  Optimal selection of reduced rank estimators of high-dimensional matrices , 2010, 1004.2995.

[13]  John Shawe-Taylor,et al.  A PAC analysis of a Bayesian estimator , 1997, COLT '97.

[14]  Sumio Watanabe,et al.  Stochastic complexities of reduced rank regression in Bayesian estimation , 2005, Neural Networks.

[15]  Jean-Yves Audibert,et al.  Robust linear least squares regression , 2010, 1010.0074.

[16]  L. Carin,et al.  Nonparametric Bayesian matrix completion , 2010, 2010 IEEE Sensor Array and Multichannel Signal Processing Workshop.

[17]  Lawrence Carin,et al.  A nonparametric Bayesian model for kernel matrix completion , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[18]  Yee Whye Teh,et al.  Variational Bayesian Approach to Movie Rating Prediction , 2007, KDD 2007.

[19]  David Gross,et al.  Recovering Low-Rank Matrices From Few Coefficients in Any Basis , 2009, IEEE Transactions on Information Theory.

[20]  Aggelos K. Katsaggelos,et al.  Low-rank matrix completion by variational sparse Bayesian learning , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[21]  Arnak S. Dalalyan,et al.  Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity , 2008, Machine Learning.

[22]  Ruslan Salakhutdinov,et al.  Bayesian probabilistic matrix factorization using Markov chain Monte Carlo , 2008, ICML '08.

[23]  Herman K. van Dijk,et al.  On the Shape of the Likelihood/Posterior in Cointegration Models , 1994, Econometric Theory.

[24]  Jacques H. Dreze,et al.  Bayesian Limited Information Analysis of the Simultaneous Equations Model , 1976 .

[25]  Miki Aoyagi,et al.  Desingularization and the Generalization Error of Reduced Rank Regression in Bayesian Estimation , 2004 .

[26]  Pierre Alquier,et al.  Rank penalized estimation of a quantum system , 2012, 1206.1711.

[27]  Yihong Gong,et al.  Large-scale collaborative prediction using a nonparametric random effects model , 2009, ICML '09.

[28]  Olivier Catoni,et al.  Statistical learning theory and stochastic optimization , 2004 .

[29]  Anton Schwaighofer,et al.  Learning Gaussian processes from multiple tasks , 2005, ICML.

[30]  Karim Lounici,et al.  Pac-Bayesian Bounds for Sparse Regression Estimation with Exponential Weights , 2010, 1009.2707.

[31]  J. Geweke,et al.  Bayesian reduced rank regression in econometrics , 1996 .

[32]  Frank Kleibergen,et al.  BAYESIAN SIMULTANEOUS EQUATIONS ANALYSIS USING REDUCED RANK STRUCTURES , 1998, Econometric Theory.

[33]  Jukka Corander,et al.  Bayesian assessment of dimensionality in reduced rank regression , 2004 .

[34]  David A. McAllester.  Some PAC-Bayesian Theorems , 1998, COLT' 98.

[35]  R. Salakhutdinov,et al.  Bayesian Probabilistic Matrix Factorization: A User Frequency Analysis , 2014, 1407.7840.

[36]  A. V. D. Vaart.  Asymptotic Statistics: Delta Method , 1998 .

[37]  Luc Bauwens,et al.  Identification restrictions and posterior densities in cointegrated Gaussian VAR system , 1996 .

[38]  O. Klopp.  Rank penalized estimators for high-dimensional matrices , 2011, 1104.1244.

[39]  A. V. D. Vaart,et al.  Asymptotic Statistics: Frontmatter , 1998 .

[40]  A. Dalalyan,et al.  Sharp Oracle Inequalities for Aggregation of Affine Estimators , 2011, 1104.3969.

[41]  Padhraic Smyth,et al.  KDD Cup and workshop 2007 , 2007, SKDD.

[42]  V. Koltchinskii,et al.  Nuclear norm penalization and optimal rates for noisy low rank matrix completion , 2010, 1011.6256.

[43]  M. Yuan,et al.  Dimension reduction and coefficient estimation in multivariate linear regression , 2007 .