On the Utility of Graphics Cards to Perform Massively Parallel Simulation of Advanced Monte Carlo Methods

We present a case study on the utility of graphics cards to perform massively parallel simulation of advanced Monte Carlo methods. Graphics cards, containing multiple Graphics Processing Units (GPUs), are self-contained parallel computational devices that can be housed in conventional desktop and laptop computers and can be thought of as prototypes of the next generation of many-core processors. For certain classes of population-based Monte Carlo algorithms they offer massively parallel simulation, with the added advantage over conventional distributed multicore processors that they are cheap, easily accessible, easy to maintain, easy to code, dedicated local devices with low power consumption. On a canonical set of stochastic simulation examples including population-based Markov chain Monte Carlo methods and Sequential Monte Carlo methods, we find speedups from 35- to 500-fold over conventional single-threaded computer code. Our findings suggest that GPUs have the potential to facilitate the growth of statistical modeling into complex data-rich domains through the availability of cheap and accessible many-core computation. We believe the speedup we observe should motivate wider use of parallelizable simulation methods and greater methodological attention to their design. This article has supplementary material online.

[1]  Jun S. Liu,et al.  Blind Deconvolution via Sequential Imputations , 1995 .

[2]  Erricos John Kontoghiorghes,et al.  Handbook of Parallel Computing and Statistics , 2005 .

[3]  A. Doucet,et al.  A Tutorial on Particle Filtering and Smoothing: Fifteen years later , 2008 .

[4]  Nicholas J. Higham,et al.  Accuracy and stability of numerical algorithms, Second Edition , 2002 .

[5]  C. Robert,et al.  Computational and Inferential Difficulties with Mixture Posterior Distributions , 2000 .

[6]  M. Pitt,et al.  Time Varying Covariances: A Factor Stochastic Volatility Approach (with discussion , 1998 .

[7]  L. Tierney.  Markov Chains for Exploring Posterior Distributions , 1994 .

[8]  Nicholas J. Higham,et al.  INVERSE PROBLEMS NEWSLETTER , 1991 .

[9]  Roman Holenstein,et al.  Particle Markov chain Monte Carlo , 2009 .

[10]  Michael A. West,et al.  Combined Parameter and State Estimation in Simulation-Based Filtering , 2001, Sequential Monte Carlo Methods in Practice.

[11]  Radford M. Neal.  Annealed importance sampling , 1998, Stat. Comput..

[12]  Robert B. Gramacy,et al.  Importance tempering , 2007, Stat. Comput..

[13]  Nicholas J. Higham,et al.  The Accuracy of Floating Point Summation , 1993, SIAM J. Sci. Comput..

[14]  C. Geyer.  Markov Chain Monte Carlo Maximum Likelihood , 1991 .

[15]  Simon P. Wilson,et al.  Parallel algorithms for Markov chain Monte Carlo methods in latent spatial Gaussian models , 2004, Stat. Comput..

[16]  Hoon Kim,et al.  Monte Carlo Statistical Methods , 2000, Technometrics.

[17]  Geoffrey J. McLachlan,et al.  Finite Mixture Models , 2019, Annual Review of Statistics and Its Application.

[18]  Anthony Brockwell.  Parallel Markov chain Monte Carlo Simulation by Pre-Fetching , 2006 .

[19]  Ajay Jasra,et al.  Markov Chain Monte Carlo Methods and the Label Switching Problem in Bayesian Mixture Modeling , 2005 .

[20]  K. Hukushima,et al.  Exchange Monte Carlo Method and Application to Spin Glass Simulations , 1995, cond-mat/9512035.

[21]  Nando de Freitas,et al.  Sequential Monte Carlo Methods in Practice , 2001, Statistics for Engineering and Information Science.

[22]  Klaus Schulten,et al.  Accelerating Molecular Modeling Applications with GPU Computing , 2009 .

[23]  P. Moral,et al.  Sequential Monte Carlo samplers , 2002, cond-mat/0212648.

[24]  Ajay Jasra,et al.  Bayesian inference for mixture models via Monte Carlo computation , 2006 .

[25]  G. Crooks.  Nonequilibrium Measurements of Free Energy Differences for Microscopically Reversible Markovian Systems , 1998 .

[26]  Marc A. Suchard,et al.  Many-core algorithms for statistical phylogenetics , 2009, Bioinform..

[27]  G. W. Snedecor.  Statistical Methods , 1964 .

[28]  Pierre L'Ecuyer,et al.  Good Parameters and Implementations for Combined Multiple Recursive Random Number Generators , 1999, Oper. Res..

[29]  Sally Rosenthal,et al.  Parallel computing and Monte Carlo algorithms , 1999 .

[30]  Tim Hesterberg,et al.  Monte Carlo Strategies in Scientific Computing , 2002, Technometrics.

[31]  C. Predescu,et al.  The incomplete beta function law for parallel tempering sampling of classical canonical systems. , 2003, The Journal of chemical physics.

[32]  Ajay Jasra,et al.  On population-based simulation for static inference , 2007, Stat. Comput..

[33]  Pierre L'Ecuyer,et al.  An Object-Oriented Random-Number Package with Many Long Streams and Substreams , 2002, Oper. Res..

[34]  Jason Wittenberg,et al.  Clarify: Software for Interpreting and Presenting Statistical Results , 2003 .

[35]  Cliburn Chan,et al.  Understanding GPU Programming for Statistical Computation: Studies in Massively Parallel Massive Mixtures , 2010, Journal of computational and graphical statistics : a joint publication of American Statistical Association, Institute of Mathematical Statistics, Interface Foundation of North America.

[36]  Vijay S. Pande,et al.  Accelerating molecular dynamic simulation on graphics processing units , 2009, J. Comput. Chem..