On the sparse Bayesian learning of linear models

ABSTRACT This work is a re-examination of the sparse Bayesian learning (SBL) of linear regression models of Tipping (2001) in a high-dimensional setting with a sparse signal. We show that in general the SBL estimator does not recover the sparsity structure of the signal. To remedy this, we propose a hard-thresholded version of the SBL estimator that achieves, for orthogonal design matrices, the non asymptotic estimation error rate of , where n is the sample size, p is the number of regressors, σ is the regression model standard deviation, and s is the number of non zero regression coefficients. We also establish that with high probability the estimator recovers the sparsity structure of the signal. In our simulations we found that the performance of thresholded SBL depends on the strength of the signal. With a weak signal thresholded SBL performs poorly compared to least absolute shrinkage and selection operator (lasso) (Tibshirani, 1996), but outperforms lasso when the signal is strong.