Adaptive Hessian-Based Nonstationary Gaussian Process Response Surface Method for Probability Density Approximation with Application to Bayesian Solution of Large-Scale Inverse Problems

We develop an adaptive Hessian-based non-stationary Gaussian process (GP) response surface method for approximating a probability density function (pdf) that exploits its structure, particularly the Hessian of its negative logarithm. Of particular interest to us are pdfs that arise from the Bayesian solution of large-scale inverse problems, which imply very expensive-to-evaluate pdfs. The method can be considered as a piecewise adaptive Gaussian approximation in which a Gaussian tailored to the local Hessian of the negative log probability density is constructed for each subregion in high dimensional parameter space. The task of efficiently partitioning the parameter space into subregions is done implicitly through Hessian-informed membership probability functions. The GP machinery is then employed to glue all local Gaussian approximations into a global analytical response surface that is far cheaper to evaluate than the original expensive probability density. The resulting response surface is also equipp...

[1]  J. Hesthaven,et al.  Nodal Discontinuous Galerkin Methods: Algorithms, Analysis, and Applications , 2007 .

[2]  O. Ghattas,et al.  A Newton-CG method for large-scale three-dimensional elastic full-waveform seismic inversion , 2008 .

[3]  Karen Willcox,et al.  Parameter and State Model Reduction for Large-Scale Statistical Inverse Problems , 2010, SIAM J. Sci. Comput..

[4]  David J. C. MacKay,et al.  Information-Based Objective Functions for Active Data Selection , 1992, Neural Computation.

[5]  D. Higdon,et al.  Computer Model Calibration Using High-Dimensional Output , 2008 .

[6]  Andreas Krause,et al.  Near-Optimal Sensor Placements in Gaussian Processes: Theory, Efficient Algorithms and Empirical Studies , 2008, J. Mach. Learn. Res..

[7]  W. Dunsmuir,et al.  Estimation of nonstationary spatial covariance structure , 2002 .

[8]  A. O'Hagan,et al.  Bayesian inference for non‐stationary spatial covariance structure via spatial deformations , 2003 .

[9]  Omar Ghattas,et al.  Analysis of the Hessian for inverse scattering problems: II. Inverse medium scattering of acoustic waves , 2012 .

[10]  Anthony O'Hagan,et al.  Uncertainty in prior elicitations: a nonparametric approach , 2007 .

[11]  Robert B. Gramacy,et al.  Parameter space exploration with Gaussian process trees , 2004, ICML.

[12]  A. O'Hagan,et al.  Bayesian calibration of computer models , 2001 .

[13]  George Biros,et al.  Parallel Lagrange-Newton-Krylov-Schur Methods for PDE-Constrained Optimization. Part I: The Krylov-Schur Solver , 2005, SIAM J. Sci. Comput..

[14]  Nicholas Zabaras,et al.  A Bayesian approach to multiscale inverse problems using the sequential Monte Carlo method , 2011 .

[15]  Geoffrey E. Hinton,et al.  Bayesian Learning for Neural Networks , 1995 .

[16]  Bui-Thanh Tan,et al.  Model-Constrained Optimization Methods for Reduction of Parameterized Large-Scale Systems , 2007 .

[17]  Tiankai Tu,et al.  High Resolution Forward And Inverse Earthquake Modeling on Terascale Computers , 2003, ACM/IEEE SC 2003 Conference (SC'03).

[18]  B. Mallick,et al.  Analyzing Nonstationary Spatial Data Using Piecewise Gaussian Processes , 2005 .

[19]  R. H. Wilcox.  Adaptive control processes—A guided tour, by Richard Bellman, Princeton University Press, Princeton, New Jersey, 1961, 255 pp., $6.50 , 1961 .

[20]  Gregory Piatetsky-Shapiro,et al.  High-Dimensional Data Analysis: The Curses and Blessings of Dimensionality , 2000 .

[21]  Timothy W. Simpson,et al.  Metamodels for Computer-based Engineering Design: Survey and recommendations , 2001, Engineering with Computers.

[22]  Bart G. van Bloemen Waanders,et al.  Fast Algorithms for Bayesian Uncertainty Quantification in Large-Scale Linear Inverse Problems Based on Low-Rank Partial Hessian Approximations , 2011, SIAM J. Sci. Comput..

[23]  Achille Messac,et al.  Extended Radial Basis Functions: More Flexible and Effective Metamodeling , 2004 .

[24]  David W. Scott,et al.  Multivariate Density Estimation: Theory, Practice, and Visualization , 1992, Wiley Series in Probability and Statistics.

[25]  C. Rasmussen,et al.  Nonstationary Gaussian Process Regression using a Latent Extension of the Input Space , 2006 .

[26]  M. C. Delfour,et al.  Shapes and Geometries - Metrics, Analysis, Differential Calculus, and Optimization, Second Edition , 2011, Advances in design and control.

[27]  Omar Ghattas,et al.  Analysis of the Hessian for inverse scattering problems: I. Inverse shape scattering of acoustic waves , 2012 .

[28]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[29]  David A. Cohn,et al.  Neural Network Exploration Using Optimal Experiment Design , 1993, NIPS.

[30]  David J. C. MacKay,et al.  Choice of Basis for Laplace Approximation , 1998, Machine Learning.

[31]  C. Reinsch.  Smoothing by spline functions , 1967 .

[32]  Christopher J Paciorek,et al.  Spatial modelling using a new class of nonstationary covariance functions , 2006, Environmetrics.

[33]  D. W. Scott,et al.  Multivariate Density Estimation, Theory, Practice and Visualization , 1992 .

[34]  D. B. P. Huynh,et al.  Certified Reduced Basis Model Characterization: a Frequentistic Uncertainty Framework , 2011 .

[35]  Karen Willcox,et al.  Model Reduction for Large-Scale Systems with High-Dimensional Parametric Input Space , 2008, SIAM J. Sci. Comput..

[36]  A. Kirsch,et al.  Two methods for solving the inverse acoustic scattering problem , 1988 .

[37]  David Higdon,et al.  Non-Stationary Spatial Modeling , 2022, 2212.08043.

[38]  Y. Maday,et al.  Reduced Basis Techniques for Stochastic Problems , 2010, 1004.0357.

[39]  P. Moral,et al.  Sequential Monte Carlo samplers , 2002, cond-mat/0212648.

[40]  P. Guttorp,et al.  Nonparametric Estimation of Nonstationary Spatial Covariance Structure , 1992 .

[41]  Thomas J. Santner,et al.  The Design and Analysis of Computer Experiments , 2003, Springer Series in Statistics.

[42]  David Galbally,et al.  Nonlinear model reduction for uncertainty quantification in large-scale inverse problems : application to nonlinear convection-diffusion-reaction equation , 2008 .

[43]  Sung-Hyuk Cha.  Comprehensive Survey on Distance/Similarity Measures between Probability Density Functions , 2007 .

[44]  Stephan R. Sain,et al.  Multi-dimensional Density Estimation , 2004 .

[45]  Anthony T. Patera,et al.  Certified reduced basis model validation: A frequentistic uncertainty framework , 2012 .

[46]  Matthew MacDonald,et al.  Shapes and Geometries , 1987 .

[47]  Martin D. Buhmann,et al.  Radial Basis Functions: Theory and Implementations: Preface , 2003 .

[48]  Montserrat Fuentes,et al.  A new class of nonstationaryspatial models , 2001 .

[49]  Habib N. Najm,et al.  Dimensionality reduction and polynomial chaos acceleration of Bayesian inference in inverse problems , 2008, J. Comput. Phys..

[50]  E. M. Wright,et al.  Adaptive Control Processes: A Guided Tour , 1961, The Mathematical Gazette.

[51]  George Biros,et al.  Parallel Lagrange-Newton-Krylov-Schur Methods for PDE-Constrained Optimization. Part II: The Lagrange-Newton Solver and Its Application to Optimal Control of Steady Viscous Flows , 2005, SIAM J. Sci. Comput..

[52]  E. Somersalo,et al.  Statistical and computational inverse problems , 2004 .

[53]  Daniela Calvetti,et al.  Introduction to Bayesian Scientific Computing: Ten Lectures on Subjective Computing , 2007 .

[54]  David J. C. MacKay,et al.  Bayesian Methods for Backpropagation Networks , 1996 .

[55]  Omar Ghattas,et al.  Analysis of the Hessian for inverse scattering problems: II. Inverse medium scattering of acoustic waves , 2012 .

[56]  Klaus Obermayer,et al.  Gaussian Process Regression: Active Data Selection and Test Point Rejection , 2000, DAGM-Symposium.

[57]  David Gottlieb,et al.  On the construction and analysis of absorbing layers in CEM , 1998 .

[58]  A. OHagan,et al.  Bayesian analysis of computer code outputs: A tutorial , 2006, Reliab. Eng. Syst. Saf..

[59]  Dave Higdon,et al.  Combining Field Data and Computer Simulations for Calibration and Prediction , 2005, SIAM J. Sci. Comput..

[60]  Omar Ghattas,et al.  Analysis of the Hessian for Inverse Scattering Problems. Part III: Inverse Medium Scattering of Electromagnetic Waves in Three Dimensions , 2013 .

[61]  Nicholas Zabaras,et al.  Using Bayesian statistics in the estimation of heat source in radiation , 2005 .

[62]  Habib N. Najm,et al.  Stochastic spectral methods for efficient Bayesian solution of inverse problems , 2005, J. Comput. Phys..

[63]  A. Patera,et al.  A posteriori error bounds for reduced-basis approximations of parametrized parabolic partial differential equations , 2005 .

[64]  James Martin,et al.  A Stochastic Newton MCMC Method for Large-Scale Statistical Inverse Problems with Application to Seismic Inversion , 2012, SIAM J. Sci. Comput..

[65]  Heikki Haario,et al.  DRAM: Efficient adaptive MCMC , 2006, Stat. Comput..

[66]  J. Andrés Christen,et al.  Advances in the Design of Gaussian Processes as Surrogate Models for Computer Experiments , 2008 .

[67]  Stefan Ulbrich,et al.  Optimization with PDE Constraints , 2008, Mathematical modelling.