Accelerating proximal Markov chain Monte Carlo by using explicit stabilised methods

We present a highly efficient proximal Markov chain Monte Carlo methodology to perform Bayesian computation in imaging problems. Similarly to previous proximal Monte Carlo approaches, the proposed method is derived from an approximation of the Langevin diffusion. However, instead of the conventional Euler-Maruyama approximation that underpins existing proximal Monte Carlo methods, here we use a state-of-the-art orthogonal Runge-Kutta-Chebyshev stochastic approximation that combines several gradient evaluations to significantly accelerate its convergence speed, similarly to accelerated gradient optimisation methods. The proposed methodology is demonstrated via a range of numerical experiments, including non-blind image deconvolution, hyperspectral unmixing, and tomographic reconstruction, with total-variation and $\ell_1$-type priors. Comparisons with Euler-type proximal Monte Carlo methods confirm that the Markov chains generated with our method exhibit significantly faster convergence speeds, achieve larger effective sample sizes, and produce lower mean square estimation errors at equal computational budget.

[1]  Amir Beck,et al.  First-Order Methods in Optimization , 2017 .

[2]  A. Abdulle Explicit stabilized Runge-Kutta methods , 2011 .

[3]  Nicolas Dobigeon,et al.  SPARSE BAYESIAN BINARY LOGISTIC REGRESSION USING THE SPLIT-AND-AUGMENTED GIBBS SAMPLER , 2018, 2018 IEEE 28th International Workshop on Machine Learning for Signal Processing (MLSP).

[4]  É. Moulines,et al.  Non-asymptotic convergence analysis for the Unadjusted Langevin Algorithm , 2015, 1507.05021.

[5]  Marcelo Pereyra,et al.  Maximum-a-Posteriori Estimation with Bayesian Confidence Regions , 2016, SIAM J. Imaging Sci..

[6]  J. Mixter Fast , 2012 .

[7]  Christian P. Robert,et al.  Bayesian computation: a summary of the current state, and samples backwards and forwards , 2015, Statistics and Computing.

[8]  Assyr Abdulle,et al.  High Order Numerical Approximation of the Invariant Measure of Ergodic SDEs , 2014, SIAM J. Numer. Anal..

[9]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[10]  Eric Moulines,et al.  Efficient Bayesian Computation by Proximal Markov Chain Monte Carlo: When Langevin Meets Moreau , 2016, SIAM J. Imaging Sci..

[11]  Veit Elser,et al.  Benchmark problems for phase retrieval , 2017, SIAM J. Imaging Sci..

[12]  Amel Benazza-Benyahia,et al.  Majorize-Minimize adapted metropolis-hastings algorithm. Application to multichannel image recovery , 2014, 2014 22nd European Signal Processing Conference (EUSIPCO).

[13]  Marcelo Pereyra,et al.  Revisiting Maximum-A-Posteriori Estimation in Log-Concave Models , 2016, SIAM J. Imaging Sci..

[14]  M. Girolami,et al.  Riemann manifold Langevin and Hamiltonian Monte Carlo methods , 2011, Journal of the Royal Statistical Society: Series B (Statistical Methodology).

[15]  Nicolas Dobigeon,et al.  Bayesian Antisparse Coding , 2015, IEEE Transactions on Signal Processing.

[16]  Paul C. Beard,et al.  Enhancing Compressed Sensing 4D Photoacoustic Tomography by Simultaneous Motion Estimation , 2018, SIAM J. Imaging Sci..

[17]  Yair Carmon,et al.  Accelerated Methods for NonConvex Optimization , 2018, SIAM J. Optim..

[18]  D. Higham A-Stability and Stochastic Mean-Square Stability , 2000 .

[19]  Alain Durmus,et al.  Analysis of Langevin Monte Carlo via Convex Optimization , 2018, J. Mach. Learn. Res..

[20]  Amel Benazza-Benyahia,et al.  An auxiliary variable method for Langevin based MCMC algorithms , 2016, 2016 IEEE Statistical Signal Processing Workshop (SSP).

[21]  Simon R. Arridge,et al.  Solving inverse problems using data-driven models , 2019, Acta Numerica.

[22]  José M. Bioucas-Dias,et al.  Fast Image Recovery Using Variable Splitting and Constrained Optimization , 2009, IEEE Transactions on Image Processing.

[23]  Antonin Chambolle,et al.  An introduction to continuous optimization for imaging , 2016, Acta Numerica.

[24]  Alfred O. Hero,et al.  A Survey of Stochastic Simulation and Optimization Methods in Signal Processing , 2015, IEEE Journal of Selected Topics in Signal Processing.

[25]  José M. Bioucas-Dias,et al.  Maximum-a-posteriori estimation with unknown regularisation parameters , 2015, 2015 23rd European Signal Processing Conference (EUSIPCO).

[26]  Jeffrey A. Fessler,et al.  Fast, iterative image reconstruction for MRI in the presence of field inhomogeneities , 2003, IEEE Transactions on Medical Imaging.

[27]  I. Daubechies,et al.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint , 2003, math/0307152.

[28]  Assyr Abdulle,et al.  Optimal Explicit Stabilized Integrator of Weak Order 1 for Stiff and Ergodic Stochastic Differential Equations , 2017, SIAM/ASA J. Uncertain. Quantification.

[29]  Marcelo Pereyra,et al.  Uncertainty quantification for radio interferometric imaging: II. MAP estimation , 2017, Monthly Notices of the Royal Astronomical Society.

[30]  Marcelo Pereyra,et al.  Maximum Likelihood Estimation of Regularisation Parameters , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[31]  Haichao Zhang,et al.  Revisiting Bayesian blind deconvolution , 2013, J. Mach. Learn. Res..

[32]  T. Faniran Numerical Solution of Stochastic Differential Equations , 2015 .

[33]  Alain Durmus,et al.  Maximum likelihood estimation of regularisation parameters in high-dimensional inverse problems: an empirical Bayesian approach , 2019, 1911.11709.

[34]  Marcelo Pereyra,et al.  Proximal Markov chain Monte Carlo algorithms , 2013, Statistics and Computing.

[35]  Jean-Yves Tourneret,et al.  A Hamiltonian Monte Carlo Method for Non-Smooth Energy Sampling , 2014, IEEE Transactions on Signal Processing.

[36]  Desmond J. Higham,et al.  Mean-Square and Asymptotic Stability of the Stochastic Theta Method , 2000, SIAM J. Numer. Anal..

[37]  Andre Wibisono,et al.  A variational perspective on accelerated methods in optimization , 2016, Proceedings of the National Academy of Sciences.

[38]  Alfonso Fernández-Manso,et al.  Spectral unmixing , 2012 .

[39]  Jean-Yves Tourneret,et al.  Preconditioned P-ULA for Joint Deconvolution-Segmentation of Ultrasound Images , 2019, IEEE Signal Processing Letters.

[40]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[41]  Faming Liang,et al.  Statistical and Computational Inverse Problems , 2006, Technometrics.

[42]  Nicolas Dobigeon,et al.  Split-and-Augmented Gibbs Sampler—Application to Large-Scale Inference Problems , 2018, IEEE Transactions on Signal Processing.

[43]  Antonio J. Plaza,et al.  Total Variation Spatial Regularization for Sparse Hyperspectral Unmixing , 2012, IEEE Transactions on Geoscience and Remote Sensing.

[44]  Eric Moulines,et al.  Sampling from a log-concave distribution with compact support with proximal Langevin Monte Carlo , 2017, COLT.

[45]  R. Tweedie,et al.  Exponential convergence of Langevin distributions and their discrete approximations , 1996 .

[46]  Christian P. Robert,et al.  The Bayesian choice : from decision-theoretic foundations to computational implementation , 2007 .

[47]  Marcelo Pereyra,et al.  Comparing Bayesian models in the absence of ground truth , 2016, 2016 24th European Signal Processing Conference (EUSIPCO).

[48]  Anwesha Bhattacharyya,et al.  An approach to large-scale Quasi-Bayesian inference with spike-and-slab priors , 2018 .

[49]  Daniel Paulin,et al.  Efficient MCMC Sampling with Dimension-Free Convergence Rate using ADMM-type Splitting , 2019, J. Mach. Learn. Res..

[50]  Charles Bouveyron,et al.  High-Dimensional Mixture Models for Unsupervised Image Denoising (HDMI) , 2018, SIAM J. Imaging Sci..

[51]  Charles J. Geyer,et al.  Practical Markov Chain Monte Carlo , 1992 .

[52]  Assyr Abdulle,et al.  S-ROCK: Chebyshev Methods for Stiff Stochastic Differential Equations , 2008, SIAM J. Sci. Comput..

[53]  Audrey Repetti,et al.  Scalable Bayesian Uncertainty Quantification in Imaging Inverse Problems via Convex Optimization , 2018, SIAM J. Imaging Sci..

[54]  Nicolas Dobigeon,et al.  Bayesian Image Restoration under Poisson Noise and Log-concave Prior , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[55]  Michael Elad,et al.  The Little Engine That Could: Regularization by Denoising (RED) , 2016, SIAM J. Imaging Sci..

[56]  K. Zygalakis,et al.  Explicit stabilised gradient descent for faster strongly convex optimisation , 2018, BIT Numerical Mathematics.

[57]  M. Nikolova An Algorithm for Total Variation Minimization and Applications , 2004 .

[58]  Y. Atchadé,et al.  Regularization and Computation with high-dimensional spike-and-slab posterior distributions , 2018, 1803.10282.

[59]  Jong Chul Ye,et al.  Deep Convolutional Framelets: A General Deep Learning Framework for Inverse Problems , 2017, SIAM J. Imaging Sci..