A Shrinkage-Thresholding Metropolis Adjusted Langevin Algorithm for Bayesian Variable Selection

This paper introduces a new Markov Chain Monte Carlo method for Bayesian variable selection in high dimensional settings. The algorithm is a Hastings-Metropolis sampler with a proposal mechanism which combines a Metropolis Adjusted Langevin (MALA) step to propose local moves associated with a shrinkage-thresholding step allowing to propose new models. The geometric ergodicity of this new trans-dimensional Markov Chain Monte Carlo sampler is established. An extensive numerical experiment, on simulated and real data, is presented to illustrate the performance of the proposed algorithm in comparison with some more classical trans-dimensional algorithms.

[1]  L. Breiman.  The Little Bootstrap and other Methods for Dimensionality Selection in Regression: X-Fixed Prediction Error , 1992 .

[2]  Richard L. Tweedie,et al.  Markov Chains and Stochastic Stability , 1993, Communications and Control Engineering Series.

[3]  E. George,et al.  Journal of the American Statistical Association is currently published by American Statistical Association. , 2007 .

[4]  N. L. Johnson,et al.  Continuous Univariate Distributions. , 1995 .

[5]  B. Carlin,et al.  Bayesian Model Choice Via Markov Chain Monte Carlo Methods , 1995 .

[6]  P. Green.  Reversible jump Markov chain Monte Carlo computation and Bayesian model determination , 1995 .

[7]  R. Tweedie,et al.  Rates of convergence of the Hastings and Metropolis algorithms , 1996 .

[8]  R. Tweedie,et al.  Exponential convergence of Langevin distributions and their discrete approximations , 1996 .

[9]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[10]  Hoon Kim,et al.  Monte Carlo Statistical Methods , 2000, Technometrics.

[11]  S. F. Jarner,et al.  Geometric ergodicity of Metropolis algorithms , 2000 .

[12]  T. Fearn,et al.  Bayesian Wavelet Regression on Curves With Application to a Spectroscopic Calibration Problem , 2001 .

[13]  Kevin Barraclough,et al.  I and i , 2001, BMJ : British Medical Journal.

[14]  S. Godsill.  On the Relationship Between Markov chain Monte Carlo Methods for Model Uncertainty , 2001 .

[15]  Petros Dellaportas,et al.  On Bayesian model and variable selection using MCMC , 2002, Stat. Comput..

[16]  Matthew West,et al.  Bayesian factor regression models in the''large p , 2003 .

[17]  G. Roberts,et al.  Efficient construction of reversible jump Markov chain Monte Carlo proposal distributions , 2003 .

[18]  J. S. Rao,et al.  Spike and slab variable selection: Frequentist and Bayesian strategies , 2005, math/0505633.

[19]  David J. Nott,et al.  Adaptive sampling for Bayesian variable selection , 2005 .

[20]  Y. Atchadé.  An Adaptive Version for the Metropolis Adjusted Langevin Algorithm with a Truncated Drift , 2006 .

[21]  J. Rosenthal,et al.  Harris recurrence of Metropolis-within-Gibbs and trans-dimensional Markov chains , 2006, math/0702412.

[22]  G. Roberts,et al.  OPTIMAL SCALING FOR PARTIALLY UPDATING MCMC ALGORITHMS , 2006, math/0607054.

[23]  A. Tsybakov,et al.  Sparsity oracle inequalities for the Lasso , 2007, 0705.3308.

[24]  S. Geer.  HIGH-DIMENSIONAL GENERALIZED LINEAR MODELS AND THE LASSO , 2008, 0804.0703.

[25]  G. Casella,et al.  The Bayesian Lasso , 2008 .

[26]  Arnaud Doucet,et al.  Sparse Bayesian nonparametric regression , 2008, ICML '08.

[27]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[28]  Arnak S. Dalalyan,et al.  Sparse Regression Learning by Aggregation and Langevin Monte-Carlo , 2009, COLT.

[29]  R. O’Hara,et al.  A review of Bayesian variable selection methods: what, how and which , 2009 .

[30]  P. Bickel,et al.  SIMULTANEOUS ANALYSIS OF LASSO AND DANTZIG SELECTOR , 2008, 0801.1095.

[31]  Jian Li,et al.  Efficient sparse Bayesian learning via Gibbs sampling , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[32]  J. Griffin,et al.  BAYESIAN HYPER‐LASSOS WITH NON‐CONVEX PENALIZATION , 2011 .

[33]  D. Dunson,et al.  Bayesian Variable Selection via Particle Stochastic Search. , 2011, Statistics & probability letters.

[34]  Bhaskar D. Rao,et al.  Latent Variable Bayesian Models for Promoting Sparsity , 2011, IEEE Transactions on Information Theory.

[35]  F. Lucka.  Fast Markov chain Monte Carlo sampling for sparse Bayesian inference in high-dimensional inverse problems using L1-type priors , 2012, 1206.0262.

[36]  A. Tsybakov,et al.  Sparse Estimation by Exponential Weighting , 2011, 1108.5116.

[37]  Kai Siedenburg.  PERSISTENT EMPIRICAL WIENER ESTIMATION WITH ADAPTIVE THRESHOLD SELECTION FOR AUDIO DENOISING , 2012 .

[38]  Christophe Andrieu,et al.  Annealed Importance Sampling Reversible Jump MCMC Algorithms , 2013 .

[39]  Nicolas Chopin,et al.  Sequential Monte Carlo on large binary sampling spaces , 2011, Statistics and Computing.

[40]  P. Dellaportas,et al.  An MCMC model search algorithm for regression problems , 2013 .

[41]  S. Schmidler,et al.  Adaptive Markov Chain Monte Carlo for Bayesian Variable Selection , 2013 .

[42]  Jim E. Griffin,et al.  Adaptive Monte Carlo for Bayesian Variable Selection in Regression Models , 2013 .

[43]  Marcelo Pereyra,et al.  Proximal Markov chain Monte Carlo algorithms , 2013, Statistics and Computing.

[44]  G. Malsiner‐Walli,et al.  Comparing Spike and Slab Priors for Bayesian Variable Selection , 2016, 1812.07259.