Integrals over Gaussians under Linear Domain Constraints

Integrals of linearly constrained multivariate Gaussian densities are a frequent problem in machine learning and statistics, arising in tasks like generalized linear models and Bayesian optimization. Yet they are notoriously hard to compute, and to further complicate matters, the numerical values of such integrals may be very small. We present an efficient black-box algorithm that exploits geometry for the estimation of integrals over a small, truncated Gaussian volume, and to simulate therefrom. Our algorithm uses the Holmes-Diaconis-Ross (HDR) method combined with an analytic version of elliptical slice sampling (ESS). Adapted to the linear setting, ESS allows for rejection-free sampling, because intersections of ellipses and domain boundaries have closed-form solutions. The key idea of HDR is to decompose the integral into easier-to-compute conditional probabilities by using a sequence of nested domains. Remarkably, it allows for direct computation of the logarithm of the integral value and thus enables the computation of extremely small probability masses. We demonstrate the effectiveness of our tailored combination of HDR and ESS on high-dimensional integrals and on entropy search for Bayesian optimization.

[1]  John Geweke,et al.  Efficient Simulation from the Multivariate Normal and Student-t Distributions Subject to Linear Constraints and the Evaluation of Constraint Probabilities , 1991 .

[2]  Subhashis Ghosal,et al.  Bayesian Inference in Nonparanormal Graphical Models , 2018, Bayesian Analysis.

[3]  Peter I. Frazier,et al.  Parallel Bayesian Global Optimization of Expensive Functions , 2016, Oper. Res..

[4]  A. Genz,et al.  Computation of Multivariate Normal and t Probabilities , 2009 .

[5]  Alan Genz,et al.  Numerical computation of rectangular bivariate and trivariate normal and t probabilities , 2004, Stat. Comput..

[6]  F. Lindgren,et al.  Excursion and contour uncertainty regions for latent Gaussian models , 2012, 1211.3946.

[7]  Andreas Scheidegger,et al.  Parameter estimation of hydrologic models using a likelihood function for censored and binary observations. , 2017, Water research.

[8]  Anthony J. Hayter,et al.  The evaluation of general non‐centred orthant probabilities , 2003 .

[9]  Yuh-Ing Chen,et al.  Identification of the minimum effective dose for right-censored survival data , 2007, Comput. Stat. Data Anal..

[10]  Anthony J. Hayter,et al.  The evaluation of trivariate normal probabilities defined by linear inequalities , 2013 .

[11]  P. Moral,et al.  Sequential Monte Carlo samplers , 2002, cond-mat/0212648.

[12]  Patrick J. Laub,et al.  Efficient Simulation for Dependent Rare Events with Applications to Extremes , 2016, 1609.09725.

[13]  Robert E. Melchers,et al.  Structural Reliability: Analysis and Prediction , 1987 .

[14]  Jaime F. Fisac,et al.  A General Safety Framework for Learning-Based Control in Uncertain Robotic Systems , 2017, IEEE Transactions on Automatic Control.

[15]  David Ginsbourger,et al.  Estimating Orthant Probabilities of High-Dimensional Gaussian Vectors with An Application to Set Estimation , 2016, 1603.05031.

[16]  Stephan R. Sain,et al.  Spatio-temporal exceedance locations and confidence regions , 2013, 1311.7257.

[17]  Noboru Nomura,et al.  Evaluation of Gaussian orthant probabilities based on orthogonal projections to subspaces , 2016, Stat. Comput..

[18]  Olivier Roustant,et al.  Finite-dimensional Gaussian approximation with linear inequality constraints , 2017, SIAM/ASA J. Uncertain. Quantification.

[19]  Matthew W. Hoffman,et al.  Predictive Entropy Search for Efficient Global Optimization of Black-box Functions , 2014, NIPS.

[20]  Susan Holmes,et al.  Three Examples of Monte-Carlo Markov Chains: At the Interface Between Statistical Computing, Computer Science, and Statistical Mechanics , 1995 .

[21]  C. S. Manohar,et al.  Markov chain splitting methods in structural reliability integral estimation , 2015 .

[22]  J. Cunningham,et al.  Gaussian Probabilities and Expectation Propagation , 2011, 1111.6832.

[23]  Mingyuan Zhou,et al.  Fast Simulation of Hyperplane-Truncated Multivariate Normal Distributions , 2016, 1607.04751.

[24]  Noboru Nomura.  Computation of multivariate normal probabilities with polar coordinate systems , 2014 .

[25]  Dirk P. Kroese,et al.  Handbook of Monte Carlo Methods , 2011 .

[26]  Z. Botev.  The normal law under linear restrictions: simulation and estimation via minimax tilting , 2016, 1603.04166.

[27]  Christopher K. I. Williams,et al.  Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning) , 2005 .

[28]  Pierre Del Moral,et al.  Sequential Monte Carlo for rare event estimation , 2012, Stat. Comput..

[29]  Ling Li,et al.  Bayesian Subset Simulation , 2016, SIAM/ASA J. Uncertain. Quantification.

[30]  Radford M. Neal.  Slice Sampling , 2003, The Annals of Statistics.

[31]  Vijayan N. Nair,et al.  Bayesian Inference for Multivariate Ordinal Data Using Parameter Expansion , 2008, Technometrics.

[32]  Ryan P. Adams,et al.  Elliptical slice sampling , 2009, AISTATS.

[33]  A. Davison,et al.  Composite likelihood estimation for the Brown–Resnick process , 2013 .

[34]  Wim P. M. Vijverberg,et al.  Monte Carlo evaluation of multivariate normal probabilities , 1997 .

[35]  Axel Gandy,et al.  Fast computation of high-dimensional multivariate normal probabilities , 2011, Comput. Stat. Data Anal..

[36]  David E. Keyes,et al.  Hierarchical Decompositions for the Computation of High-Dimensional Multivariate Normal Probabilities , 2018 .

[37]  Alexander I. J. Forrester,et al.  Multi-fidelity optimization via surrogate modelling , 2007, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[38]  Hélène Jacqmin-Gadda,et al.  Mixed models for longitudinal left-censored repeated measures , 2004, Comput. Methods Programs Biomed..

[39]  Nando de Freitas,et al.  Taking the Human Out of the Loop: A Review of Bayesian Optimization , 2016, Proceedings of the IEEE.

[40]  Hui Li,et al.  Quadratically gated mixture of experts for incomplete data classification , 2007, ICML '07.

[41]  Daniel Straub,et al.  Reliability analysis of deteriorating structural systems , 2020 .

[42]  J. Beck,et al.  Estimation of Small Failure Probabilities in High Dimensions by Subset Simulation , 2001 .

[43]  Anthony J. Hayter,et al.  The evaluation of two-sided orthant probabilities for a quadrivariate normal distribution , 2012, Comput. Stat..

[44]  Philipp Hennig,et al.  Entropy Search for Information-Efficient Global Optimization , 2011, J. Mach. Learn. Res..

[45]  A. Genz.  Numerical Computation of Multivariate Normal Probabilities , 1992 .

[46]  Andrés F. López-Lopera,et al.  Gaussian Process Modulated Cox Processes under Linear Inequality Constraints , 2019, AISTATS.

[47]  J. Ashford,et al.  Multi-variate probit analysis. , 1970, Biometrics.

[48]  J. Beck,et al.  First excursion probabilities for linear systems by very efficient importance sampling , 2001 .

[49]  Hillary Koch,et al.  Fast and Exact Simulation of Multivariate Normal and Wishart Random Variables with Box Constraints , 2019, 1907.00057.

[50]  Marc G. Genton,et al.  On the likelihood function of Gaussian max-stable processes , 2011 .

[51]  H. Joe.  Approximations to Multivariate Normal Rectangle Probabilities Based on Conditional Expectations , 1995 .

[52]  Peter S. Craig,et al.  A new reconstruction of multivariate normal orthant probabilities , 2007 .

[53]  Fredrik Lindsten,et al.  Elements of Sequential Monte Carlo , 2019, Found. Trends Mach. Learn..

[54]  Christian Agrell,et al.  Gaussian processes with linear operator inequality constraints , 2019, J. Mach. Learn. Res..

[55]  Lawrence Carin,et al.  Nonlinear Statistical Learning with Truncated Gaussian Graphical Models , 2016, ICML.

[56]  Tamás Szántai,et al.  Computing Multivariate Normal Probabilities: A New Look , 2002 .

[57]  Sébastien Da Veiga,et al.  Gaussian process modeling with inequality constraints , 2012 .