Uncertainty Quantification and Experimental Design for large-scale linear Inverse Problems under Gaussian Process Priors

We consider the use of Gaussian process (GP) priors for solving inverse problems in a Bayesian framework. As is well known, the computational complexity of GPs scales cubically in the number of datapoints. We here show that in the context of inverse problems involving integral operators, one faces additional difficulties that hinder inversion on large grids. Furthermore, in that context, covariance matrices can become too large to be stored. By leveraging results about sequential disintegrations of Gaussian measures, we are able to introduce an implicit representation of posterior covariance matrices that reduces the memory footprint by only storing low rank intermediate matrices, while allowing individual elements to be accessed on-the-fly without needing to build full posterior covariance matrices. Moreover, it allows for fast sequential inclusion of new observations. These features are crucial when considering sequential experimental design tasks. We demonstrate our approach by computing sequential data collection plans for excursion set recovery for a gravimetric inverse problem, where the goal is to provide fine resolution estimates of high density regions inside the Stromboli volcano, Italy. Sequential data collection plans are computed by extending the weighted integrated variance reduction (wIVR) criterion to inverse problems. Our results show that this criterion is able to significantly reduce the uncertainty on the excursion volume, reaching close to minimal levels of residual uncertainty. Overall, our techniques allow the advantages of probabilistic models to be brought to bear on large-scale inverse problems arising in the natural sciences. Particularly, applying the latest developments in Bayesian sequential experimental design on realistic large-scale problems opens new venues of research at a crossroads between mathematical modelling of natural phenomena, statistical data science and active learning.

[1]  Vaja Tarieladze,et al.  Disintegration of Gaussian measures and average-case optimal algorithms , 2007, J. Complex..

[2]  Gerald B. Folland,et al.  Real Analysis: Modern Techniques and Their Applications , 1984 .

[3]  David Ginsbourger,et al.  Learning excursion sets of vector-valued Gaussian random fields for autonomous ocean sampling , 2020, ArXiv.

[4]  G. Roberts,et al.  MCMC Methods for Functions: ModifyingOld Algorithms to Make Them Faster , 2012, 1202.0709.

[5]  A. Mandelbaum,et al.  Linear estimators and measurable linear transformations on a Hilbert space , 1984 .

[6]  Hyunjoong Kim,et al.  Functional Analysis I , 2017 .

[7]  X. Emery.  The kriging update equations and their application to the selection of neighboring data , 2009 .

[8]  Stephen Tyree,et al.  Exact Gaussian Processes on a Million Data Points , 2019, NeurIPS.

[9]  David Ginsbourger,et al.  Corrected Kriging Update Formulae for Batch-Sequential Data Assimilation , 2012, 1203.6452.

[10]  L. Cocchi,et al.  3-D density structure and geological evolution of Stromboli volcano (Aeolian Islands, Italy) inferred from land-based and sea-surface gravity data , 2014, 1701.01627.

[11]  Victor Picheny,et al.  Fast Parallel Kriging-Based Stepwise Uncertainty Reduction With Application to the Identification of an Excursion Set , 2014, Technometrics.

[12]  Houman Owhadi,et al.  Conditioning Gaussian measure on Hilbert space , 2015, 1506.04208.

[13]  J. Chilès,et al.  Geostatistics: Modeling Spatial Uncertainty , 1999 .

[14]  David Ginsbourger,et al.  Adaptive Design of Experiments for Conservative Estimation of Excursion Sets , 2016, Technometrics.

[15]  J. Bect,et al.  A supermartingale approach to Gaussian process based sequential design of experiments , 2016, Bernoulli.

[16]  David Williams,et al.  Probability with Martingales , 1991, Cambridge mathematical textbooks.

[17]  Andrew M. Stuart,et al.  Inverse problems: A Bayesian perspective , 2010, Acta Numerica.

[18]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[19]  T. Sullivan,et al.  The linear conditional expectation in Hilbert space , 2020, 2008.12070.

[20]  Jangsun Baek,et al.  Efficient computation of maximum likelihood estimators in a spatial linear model with power exponential covariogram , 2001 .

[21]  R. Bass,et al.  Review: P. Billingsley, Convergence of probability measures , 1971 .

[22]  Haiyu Gao,et al.  The updated kriging variance and optimal sample design , 1996 .

[23]  R. Blakely.  Potential theory in gravity and magnetic applications , 1996 .

[24]  Jan Mandel,et al.  Efficient Implementation of the Ensemble Kalman Filter Efficient Implementation of the Ensemble Kalman Filter , 2022 .

[25]  Chantal de Fouquet,et al.  Reminders on the Conditioning Kriging , 1994 .

[26]  Thomas B. Schön,et al.  Linearly constrained Gaussian processes , 2017, NIPS.

[27]  Victor Picheny,et al.  Adaptive Designs of Experiments for Accurate Approximation of a Target Region , 2010 .

[28]  Keith F. Taylor,et al.  Induced Representations of Locally Compact Groups , 2013 .

[29]  H. Robbins.  On the Measure of a Random Set , 1944 .

[30]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[31]  A. Stuart,et al.  ANALYSIS OF SPDES ARISING IN PATH SAMPLING PART II: THE NONLINEAR CASE , 2006, math/0601092.

[32]  S. Gupta,et al.  GRAVITATIONAL ATTRACTION OF A RECTANGULAR PARALLELEPIPED , 1977 .

[33]  Ling Li,et al.  Sequential design of computer experiments for the estimation of a probability of failure , 2010, Statistics and Computing.

[34]  David Ginsbourger,et al.  Fast Update of Conditional Simulation Ensembles , 2015, Mathematical Geosciences.

[35]  David Ginsbourger,et al.  Quantifying Uncertainties on Excursion Sets Under a Gaussian Random Field Prior , 2015, SIAM/ASA J. Uncertain. Quantification.

[36]  A. Stuart,et al.  The Bayesian Approach to Inverse Problems , 2013, 1302.6989.

[37]  A. Tarantola,et al.  Generalized Nonlinear Inverse Problems Solved Using the Least Squares Criterion (Paper 1R1855) , 1982 .

[38]  J. Voss,et al.  Analysis of SPDEs arising in path sampling. Part I: The Gaussian case , 2005 .

[39]  A. Mantoglou,et al.  The Turning Bands Method for simulation of random fields using line generation by a spectral method , 1982 .

[40]  Simo Särkkä,et al.  Linear Operators and Stochastic Partial Differential Equations in Gaussian Process Regression , 2011, ICANN.

[41]  S. Cambanis,et al.  Gaussian Processes and Gaussian Measures , 1972 .

[42]  Peter K. Kitanidis,et al.  Compressed state Kalman filter for large systems , 2015 .

[43]  Oliver G. Ernst,et al.  Bayesian Inverse Problems and Kalman Filters , 2014 .

[44]  S. Marelli,et al.  Bayesian model inversion using stochastic spectral embedding , 2020, J. Comput. Phys..

[45]  Randal J. Barnes,et al.  Efficient updating of kriging estimates and variances , 1992 .

[46]  Marc Peter Deisenroth,et al.  Efficiently sampling functions from Gaussian process posteriors , 2020, ICML.

[47]  Stephen G Pauker,et al.  Probability Distributions , 2013, Medical decision making : an international journal of the Society for Medical Decision Making.

[48]  I. Molchanov.  Theory of Random Sets , 2005 .

[49]  Carl E. Rasmussen,et al.  Derivative Observations in Gaussian Process Models of Dynamic Systems , 2002, NIPS.

[50]  N. Vakhania,et al.  Probability Distributions on Banach Spaces , 1987 .

[51]  David Ginsbourger,et al.  Estimating and quantifying uncertainties on level sets using the Vorob'ev expectation and deviation with Gaussian process models , 2013 .

[52]  Arvind K. Saibaba,et al.  Goal-oriented optimal design of experiments for large-scale Bayesian linear inverse problems , 2018, Inverse Problems.