Black-box density function estimation using recursive partitioning

We present a novel approach to Bayesian inference and general Bayesian computation that is defined through a recursive partitioning of the sample space. It does not rely on gradients, nor require any problem-specific tuning, and is asymptotically exact for any density function with a bounded domain. The output is an approximation to the whole density function including the normalization constant, via partitions organized in efficient data structures. This allows for evidence estimation, as well as approximate posteriors that allow for fast sampling and fast evaluations of the density. It shows competitive performance to recent state-of-the-art methods on synthetic and real-world problem examples including parameter inference for gravitational-wave physics.

[1]  Ronald L. Graham,et al.  An Efficient Algorithm for Determining the Convex Hull of a Finite Planar Set , 1972, Inf. Process. Lett..

[2]  R. Kronmal,et al.  On the Alias Method for Generating Random Variables From a Discrete Distribution , 1979 .

[3]  A. C. Miller,et al.  Discrete Approximations of Probability Distributions , 1983 .

[4]  C. Geyer Markov Chain Monte Carlo Maximum Likelihood , 1991 .

[5]  A. O'Hagan,et al.  Bayes–Hermite quadrature , 1991 .

[6]  Charles J. Geyer,et al.  Practical Markov Chain Monte Carlo , 1992 .

[7]  G. Parisi,et al.  Simulated tempering: a new Monte Carlo scheme , 1992, hep-lat/9205018.

[8]  C. D. Perttunen,et al.  Lipschitzian optimization without the Lipschitz constant , 1993 .

[9]  Moon,et al.  Estimation of mutual information using kernel density estimators. , 1995, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[10]  Michael D. McCool,et al.  Probability Trees , 1997, Graphics Interface.

[11]  W. Gander,et al.  Adaptive Quadrature—Revisited , 2000 .

[12]  Carl E. Rasmussen,et al.  Bayesian Monte Carlo , 2002, NIPS.

[13]  Radford M. Neal Slice Sampling , 2003, The Annals of Statistics.

[14]  A. Kraskov,et al.  Estimating mutual information. , 2003, Physical review. E, Statistical, nonlinear, and soft matter physics.

[15]  Thomas Gerstner,et al.  Numerical integration using sparse grids , 2004, Numerical Algorithms.

[16]  Michael W Deem,et al.  Parallel tempering: theory, applications, and new perspectives. , 2005, Physical chemistry chemical physics : PCCP.

[17]  D. Hinkley Annals of Statistics , 2006 .

[18]  Paul V. Preckel,et al.  Gaussian cubature: A practitioner's guide , 2007, Math. Comput. Model..

[19]  Marc Levoy,et al.  Gaussian KD-trees for fast high-dimensional filtering , 2009, ACM Trans. Graph..

[20]  Andrew Adams,et al.  Fast High‐Dimensional Filtering Using the Permutohedral Lattice , 2010, Comput. Graph. Forum.

[21]  Vladimír Lacko,et al.  On decompositional algorithms for uniform sampling from n-spheres and n-balls , 2010, J. Multivar. Anal..

[22]  Radford M. Neal MCMC Using Hamiltonian Dynamics , 2011, 1206.1901.

[23]  Carl E. Rasmussen,et al.  Active Learning of Model Evidence Using Bayesian Quadrature , 2012, NIPS.

[24]  Jinyu Wen,et al.  A discrete point estimate method for probabilistic load flow based on the measured data of wind power , 2012, 2012 IEEE Industry Applications Society Annual Meeting.

[25]  Hui Jiang,et al.  Multivariate Density Estimation by Bayesian Sequential Partitioning , 2013 .

[26]  Andrew Gordon Wilson,et al.  Gaussian Process Kernels for Pattern Discovery and Extrapolation , 2013, ICML.

[27]  Alexis Akira Toda,et al.  Discrete approximations of continuous distributions by maximum entropy , 2013 .

[28]  Chong Wang,et al.  Stochastic variational inference , 2012, J. Mach. Learn. Res..

[29]  Sean Gerrish,et al.  Black Box Variational Inference , 2013, AISTATS.

[30]  Ken'ichiro Tanaka,et al.  Discretizing Distributions with Exact Moments: Error Estimate and Convergence Analysis , 2015, SIAM J. Numer. Anal..

[31]  Michael A. Osborne,et al.  Probabilistic numerics and uncertainty in computations , 2015, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[32]  P. Graff,et al.  Parameter estimation for compact binaries with ground-based gravitational-wave observations using the LALInference software library , 2014, 1409.7215.

[33]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[34]  Kun Yang,et al.  Density Estimation via Discrepancy Based Adaptive Sequential Partition , 2016, NIPS.

[35]  David M. Blei,et al.  Variational Inference: A Review for Statisticians , 2016, ArXiv.

[36]  Nuwan S. Ferdinand,et al.  Low-Dimensional Shaping for High-Dimensional Lattice Codes , 2016, IEEE Transactions on Wireless Communications.

[37]  Alexis Akira Toda,et al.  Discretizing Nonlinear, Non-Gaussian Markov Processes with Exact Conditional Moments , 2016 .

[38]  Yosuke Nakanishi,et al.  Optimal Sizing of Energy Storage Devices in Isolated Wind-Diesel Systems Considering Load Growth Uncertainty , 2018, IEEE Transactions on Industry Applications.

[39]  Luigi Acerbi,et al.  Variational Bayesian Monte Carlo , 2018, NeurIPS.

[40]  Yingcun Xia,et al.  Jackknife approach to the estimation of mutual information , 2018, Proceedings of the National Academy of Sciences.

[41]  P. Lasky,et al.  Bilby: A User-friendly Bayesian Inference Library for Gravitational-wave Astronomy , 2018, The Astrophysical Journal Supplement Series.

[42]  P. Lasky,et al.  Measuring the neutron star equation of state with gravitational waves: The first forty binary neutron star merger observations , 2019, Physical Review D.

[43]  B. A. Boom,et al.  GWTC-1: A Gravitational-Wave Transient Catalog of Compact Binary Mergers Observed by LIGO and Virgo during the First and Second Observing Runs , 2018 .

[44]  Jennifer A. Hoeting,et al.  Approximate Bayesian Inference via Sparse grid Quadrature Evaluation for Hierarchical Models , 2019, 1904.07270.

[45]  Edward Higson,et al.  Dynamic nested sampling: an improved algorithm for parameter estimation and evidence calculation , 2017, Statistics and Computing.

[46]  Florian T. Pokorny,et al.  Voronoi Boundary Classification: A High-Dimensional Geometric Approach via Weighted Monte Carlo Integration , 2019, ICML.

[47]  V. Raymond,et al.  Parameter estimation with a spinning multimode waveform model , 2019, Physical Review D.

[48]  P. K. Panda,et al.  GW190412: Observation of a binary-black-hole coalescence with asymmetric masses , 2020 .

[49]  J. Speagle dynesty: a dynamic nested sampling package for estimating Bayesian posteriors and evidences , 2019, Monthly Notices of the Royal Astronomical Society.

[50]  I. Mandel,et al.  An Alternative Interpretation of GW190412 as a Binary Black Hole Merger with a Rapidly Spinning Secondary , 2020, The Astrophysical Journal.

[51]  Aki Vehtari,et al.  Batch simulations and uncertainty quantification in Gaussian process surrogate approximate Bayesian computation , 2019, UAI.