Bayesian Variable Selection and Estimation for Group Lasso

The paper revisits the Bayesian group lasso and uses spike and slab priors for group variable selection. In the process, the connection of our model with penalized regression is demonstrated, and the role of posterior median for thresholding is pointed out. We show that the posterior median estimator has the oracle property for group variable selection and estimation under orthogonal designs, while the group lasso has suboptimal asymptotic estimation rate when variable selection consistency is achieved. Next we consider bi-level selection problem and propose the Bayesian sparse group selection again with spike and slab priors to select variables both at the group level and also within a group. We demonstrate via simulation that the posterior median estimator of our spike and slab models has excellent performance for both variable selection and estimation.

[1]  D. Lindley.  A STATISTICAL PARADOX , 1957 .

[2]  T. J. Mitchell,et al.  Bayesian Variable Selection in Linear Regression , 1988 .

[3]  J. Geweke,et al.  Variable selection and model comparison in regression , 1994 .

[4]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[5]  E. George,et al.  APPROACHES FOR BAYESIAN VARIABLE SELECTION , 1997 .

[6]  B. Silverman,et al.  Wavelet thresholding via a Bayesian approach , 1998 .

[7]  C. Geyer,et al.  Geometric Ergodicity of Gibbs and Block Gibbs Samplers for a Hierarchical Random Effects Model , 1998 .

[8]  Adrian E. Raftery,et al.  Bayesian model averaging: a tutorial (with comments by M. Clyde, David Draper and E. I. George, and a rejoinder by the authors , 1999 .

[9]  Wenjiang J. Fu,et al.  Asymptotics for lasso-type estimators , 2000 .

[10]  G. Casella.  Empirical Bayes Gibbs sampling. , 2001, Biostatistics.

[11]  T. Fearn,et al.  Bayes model averaging with selection of regressors , 2002 .

[12]  D. Dunson,et al.  Random Effects Selection in Linear Mixed Models , 2003, Biometrics.

[13]  R. Tibshirani,et al.  Least angle regression , 2004, math/0406456.

[14]  J. Berger,et al.  Optimal predictive model selection , 2004, math/0406464.

[15]  I. Johnstone,et al.  Needles and straw in haystacks: Empirical Bayes estimates of possibly sparse sequences , 2004, math/0410088.

[16]  M. Yuan,et al.  Efficient Empirical Bayes Variable Selection and Estimation in Linear Models , 2005 .

[17]  R. Tibshirani,et al.  Sparsity and smoothness via the fused lasso , 2005 .

[18]  H. Zou,et al.  Regularization and variable selection via the elastic net , 2005 .

[19]  G. Wahba,et al.  A NOTE ON THE LASSO AND RELATED PROCEDURES IN MODEL SELECTION , 2006 .

[20]  A. P. Dawid,et al.  Bayesian Model Averaging and Model Search Strategies , 2007 .

[21]  Hansheng Wang,et al.  Computational Statistics and Data Analysis a Note on Adaptive Group Lasso , 2022 .

[22]  A. Rinaldo,et al.  On the asymptotic properties of the group lasso estimator for linear models , 2008 .

[23]  G. Casella,et al.  The Bayesian Lasso , 2008 .

[24]  Volker Roth,et al.  The Bayesian group-Lasso for analyzing contingency tables , 2009, ICML '09.

[25]  Guillermo Sapiro,et al.  Non-Parametric Bayesian Dictionary Learning for Sparse Image Representations , 2009, NIPS.

[26]  G. Casella,et al.  Penalized regression, standard errors, and Bayesian lassos , 2010 .

[27]  Julien Mairal,et al.  Network Flow Algorithms for Structured Sparsity , 2010, NIPS.

[28]  Francesco C Stingo,et al.  INCORPORATING BIOLOGICAL INFORMATION INTO LINEAR MODELS: A BAYESIAN APPROACH TO THE SELECTION OF PATHWAYS AND GENES. , 2011, The annals of applied statistics.

[29]  S. Lahiri,et al.  Bootstrapping Lasso Estimators , 2011 .

[30]  Charles Soussen,et al.  From Bernoulli–Gaussian Deconvolution to Sparse Signal Restoration , 2011, IEEE Transactions on Signal Processing.

[31]  Kim-Anh Do,et al.  Bayesian ensemble methods for survival prediction in gene expression data , 2011, Bioinform..

[32]  Julien Mairal,et al.  Proximal Methods for Hierarchical Sparse Coding , 2010, J. Mach. Learn. Res..

[33]  A. V. D. Vaart,et al.  Needles and Straw in a Haystack: Posterior concentration for possibly sparse sequences , 2012, 1211.1197.

[34]  Jian Huang,et al.  A Selective Review of Group Selection in High-Dimensional Models. , 2012, Statistical science : a review journal of the Institute of Mathematical Statistics.

[35]  Jim E. Griffin,et al.  Structuring shrinkage: some correlated priors for regression , 2012 .

[36]  Noah Simon,et al.  A Sparse-Group Lasso , 2013 .

[37]  Ioannis Ntzoufras,et al.  On Bayesian lasso variable selection and the specification of the shrinkage parameter , 2012, Stat. Comput..

[38]  J. Griffin,et al.  Some Priors for Sparse Regression Modelling , 2013 .

[39]  Lin Zhang,et al.  Bayesian hierarchical structured variable selection methods with application to molecular inversion probe studies in breast cancer , 2014 .

[40]  B. Mallick.  VARIABLE SELECTION FOR REGRESSION MODELS , 2016 .

[41]  Adrian E. Raftery,et al.  Bayesian Model Averaging: A Tutorial , 2016 .