Design and Analysis of Efficient Parallel Bayesian Model Comparison Algorithms

A common task in science and engineering is evaluating how well a mathematical model describes a set of observations. Bayesian model comparison provides a rational and consistent method for applying logic and probability to the problem of evaluating models. Model comparison requires numerical techniques that are usually very time consuming to run. This dissertation proposes extensions to several existing numerical model comparison techniques, including nested sampling and thermodynamic integration, that incorporate parallel algorithm design to achieve significant speed-ups. Serial computer performance gains have slowed in recent years, and most processing speed improvements are seen in the area of parallel architectures. This work discusses the design, theoretical analysis, and empirical analysis of these algorithms, focusing on the performance of these algorithms with respect to accuracy and run time. Many disciplines in science and engineering make use of existing model comparison techniques. This work aims to save investigators in these disciplines time, and potentially attract those who may have been put off by time complexity concerns, by developing a general approach to model comparison that takes full advantage of modern parallel computing platforms.

[1]  A. Pettitt,et al.  Marginal likelihood estimation via power posteriors , 2008 .

[2]  E. T. Jaynes,et al.  How Does the Brain Do Plausible Reasoning , 1988 .

[3]  Rong Chen,et al.  A Theoretical Framework for Sequential Importance Sampling with Resampling , 2001, Sequential Monte Carlo Methods in Practice.

[4]  J. Aczél,et al.  Lectures on Functional Equations and Their Applications , 1968 .

[5]  G. Pólya Mathematics and Plausible Reasoning , 1958 .

[6]  P. Green Reversible jump Markov chain Monte Carlo computation and Bayesian model determination , 1995 .

[7]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[8]  A. Khachaturyan,et al.  Statistical mechanics approach to the structure determination of a crystal , 1985 .

[9]  Y. Ogata A Monte Carlo method for high dimensional integration , 1989 .

[10]  James M. Kang,et al.  Space-Filling Curves , 2017, Encyclopedia of GIS.

[11]  Sheldon M. Ross,et al.  Introduction to probability models , 1975 .

[12]  Brendon J. Brewer,et al.  Diffusive nested sampling , 2009, Stat. Comput..

[13]  W. K. Hastings,et al.  Monte Carlo Sampling Methods Using Markov Chains and Their Applications , 1970 .

[14]  J. Skilling Bayesian computation in big spaces-nested sampling and Galilean Monte Carlo , 2012 .

[15]  David L Wild,et al.  Exploring the energy landscapes of protein folding simulations with Bayesian computation. , 2010, Biophysical journal.

[16]  Radford M. Neal MCMC Using Hamiltonian Dynamics , 2011, 1206.1901.

[17]  Xiao-Li Meng,et al.  Simulating Normalizing Constants: From Importance Sampling to Bridge Sampling to Path Sampling , 1998 .

[18]  G. L. Bretthorst,et al.  Nonuniform sampling: Bandwidth and aliasing , 2001 .

[19]  Marvin H. J. Guber Bayesian Spectrum Analysis and Parameter Estimation , 1988 .

[20]  N. Metropolis,et al.  Equation of State Calculations by Fast Computing Machines , 1953, Resonance.

[21]  Mark A. Girolami,et al.  Estimating Bayes factors via thermodynamic integration and population MCMC , 2009, Comput. Stat. Data Anal..

[22]  Jiqiang Guo,et al.  Stan: A Probabilistic Programming Language. , 2017, Journal of statistical software.

[23]  Andrew Gelman,et al.  The No-U-turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo , 2011, J. Mach. Learn. Res..

[24]  Dominicus Kester,et al.  BAYESIAN INFERENCE AND MAXIMUM ENTROPY METHODS IN SCIENCE AND ENGINEERING , 2010 .

[25]  F. Feroz,et al.  MultiNest: an efficient and robust Bayesian inference tool for cosmology and particle physics , 2008, 0809.3437.

[26]  M. Tribus,et al.  Probability theory: the logic of science , 2003 .

[27]  Mark Girolami,et al.  The Controlled Thermodynamic Integral for Bayesian Model Evidence Evaluation , 2016 .

[28]  Radford M. Neal Slice Sampling , 2003, The Annals of Statistics.

[29]  Using the Z-Order Curve for Bayesian Model Comparison , 2017 .

[30]  S. Dreyfus,et al.  Thermodynamical Approach to the Traveling Salesman Problem : An Efficient Simulation Algorithm , 2004 .

[31]  A. Lasenby,et al.  polychord: next-generation nested sampling , 2015, 1506.00171.

[32]  Lei Cao,et al.  Combined-chain nested sampling for efficient Bayesian model comparison , 2017, Digit. Signal Process..

[33]  Doreen Eichel,et al.  Data Analysis A Bayesian Tutorial , 2016 .

[34]  J. Kirkwood Statistical Mechanics of Fluid Mixtures , 1935 .

[35]  Kenneth Dixon,et al.  Introduction to Stochastic Modeling , 2011 .

[36]  Zbigniew J. Czech,et al.  Introduction to Parallel Computing , 2017 .

[37]  Walter R. Gilks,et al.  Bayesian model comparison via jump diffusions , 1995 .

[38]  Radford M. Neal Probabilistic Inference Using Markov Chain Monte Carlo Methods , 2011 .

[39]  J. Skilling Nested sampling for general Bayesian computation , 2006 .

[40]  H. Tijms A First Course in Stochastic Models , 2003 .

[41]  John Skilling Programming the Hilbert curve , 2004 .

[42]  Paul M. Goggans,et al.  Parallelized nested sampling , 2014 .

[43]  P. Goggans,et al.  Using Thermodynamic Integration to Calculate the Posterior Probability in Bayesian Model Selection Problems , 2004 .

[44]  B. L. Burrows,et al.  A New Approach to Numerical Integration , 1980 .

[45]  L. Connors,et al.  The Washington Post , 2003 .

[46]  R. T. Cox Probability, frequency and reasonable expectation , 1990 .

[47]  P. Gregory Bayesian Logical Data Analysis for the Physical Sciences: The how-to of Bayesian inference , 2005 .