Goal-Oriented Optimal Approximations of Bayesian Linear Inverse Problems

We propose optimal dimensionality reduction techniques for the solution of goal-oriented linear-Gaussian inverse problems, where the quantity of interest (QoI) is a function of the inversion parameters. These approximations are suitable for large-scale applications. In particular, we study the approximation of the posterior covariance of the QoI as a low-rank negative update of its prior covariance and prove optimality of this update with respect to the natural geodesic distance on the manifold of symmetric positive definite matrices. Assuming exact knowledge of the posterior mean of the QoI, the optimality results extend to optimality in distribution with respect to the Kullback--Leibler divergence and the Hellinger distance between the associated distributions. We also propose the approximation of the posterior mean of the QoI as a low-rank linear function of the data and prove optimality of this approximation with respect to a weighted Bayes risk. Both of these optimal approximations avoid the explicit...

[1]  直樹 武川,et al.  Regularization , 2019, Encyclopedia of Continuum Mechanics.

[2]  Giovanni Guglielmini,et al.  Elementi di trasmissione del calore , 1990 .

[3]  Colin Fox,et al.  Sampling Gaussian Distributions in Krylov Spaces with Conjugate Gradients , 2012, SIAM J. Sci. Comput..

[4]  Edmond Chow,et al.  Preconditioned Krylov Subspace Methods for Sampling Multivariate Gaussian Distributions , 2014, SIAM J. Sci. Comput..

[5]  Yu Yue,et al.  Nonstationary Spatial Gaussian Markov Random Fields , 2010 .

[6]  Karen Willcox,et al.  Nonlinear Goal-Oriented Bayesian Inference: Application to Carbon Capture and Storage , 2014, SIAM J. Sci. Comput..

[7]  Daniel Kressner,et al.  An indefinite variant of LOBPCG for definite matrix pencils , 2014, Numerical Algorithms.

[8]  Bruno Sudret,et al.  Spectral likelihood expansions for Bayesian inference , 2015, J. Comput. Phys..

[9]  C. R. Rao,et al.  Information and the Accuracy Attainable in the Estimation of Statistical Parameters , 1992 .

[10]  Yingbo Hua,et al.  Generalized Karhunen-Loeve transform , 1998, IEEE Signal Processing Letters.

[11]  Dianne P. O'Leary,et al.  Designing Optimal Spectral Filters for Inverse Problems , 2011, SIAM J. Sci. Comput..

[12]  Colin Fox,et al.  Convergence in Variance of Chebyshev Accelerated Gibbs Samplers , 2014, SIAM J. Sci. Comput..

[13]  Youssef M. Marzouk,et al.  Bayesian inference with optimal maps , 2011, J. Comput. Phys..

[14]  James Martin,et al.  A Computational Framework for Infinite-Dimensional Bayesian Inverse Problems, Part II: Stochastic Newton MCMC with Application to Ice Sheet Flow Inverse Problems , 2013, SIAM J. Sci. Comput..

[15]  James Martin,et al.  A Stochastic Newton MCMC Method for Large-Scale Statistical Inverse Problems with Application to Seismic Inversion , 2012, SIAM J. Sci. Comput..

[16]  Patrick Heimbach,et al.  A Hessian-Based Method for Uncertainty Quantification in Global Ocean State Estimation , 2014, SIAM J. Sci. Comput..

[17]  E. Somersalo,et al.  Statistical inverse problems: discretization, model reduction and inverse crimes , 2007 .

[18]  Jack Dongarra,et al.  Templates for the Solution of Algebraic Eigenvalue Problems , 2000, Software, environments, tools.

[19]  Nicholas Ayache,et al.  Geometric Means in a Novel Vector Space Structure on Symmetric Positive-Definite Matrices , 2007, SIAM J. Matrix Anal. Appl..

[20]  Owe Axelsson,et al.  On the sublinear and superlinear rate of convergence of conjugate gradient methods , 2000, Numerical Algorithms.

[21]  Nathan Halko,et al.  Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions , 2009, SIAM Rev..

[22]  Julianne Chung,et al.  Computing optimal low-rank matrix approximations for image processing , 2013, 2013 Asilomar Conference on Signals, Systems and Computers.

[23]  Christian Jutten,et al.  Classification of covariance matrices using a Riemannian-based kernel for BCI applications , 2013, Neurocomputing.

[24]  Omar Ghattas,et al.  Analysis of the Hessian for inverse scattering problems: II. Inverse medium scattering of acoustic waves , 2012 .

[25]  D. O’Leary The block conjugate gradient algorithm and related methods , 1980 .

[26]  C. R. Dietrich,et al.  Fast and Exact Simulation of Stationary Gaussian Processes through Circulant Embedding of the Covariance Matrix , 1997, SIAM J. Sci. Comput..

[27]  Peter K. Kitanidis,et al.  Randomized algorithms for generalized Hermitian eigenvalue problems with application to computing Karhunen–Loève expansion , 2013, Numer. Linear Algebra Appl..

[28]  M. Hestenes,et al.  Methods of conjugate gradients for solving linear systems , 1952 .

[29]  Shun-ichi Amari,et al.  Methods of information geometry , 2000 .

[30]  Ivan Markovsky,et al.  Structured low-rank approximation and its applications , 2008, Autom..

[31]  Heikki Haario,et al.  LARGE-SCALE KALMAN FILTERING USING THE LIMITED MEMORY BFGS METHOD , 2009 .

[32]  Robert H. Halstead,et al.  Matrix Computations , 2011, Encyclopedia of Parallel Computing.

[33]  A. Klinvex,et al.  Parallel implementations of the trace minimization scheme TraceMIN for the sparse symmetric eigenvalue problem , 2013, Comput. Math. Appl..

[34]  Per-Gunnar Martinsson,et al.  Randomized algorithms for the low-rank approximation of matrices , 2007, Proceedings of the National Academy of Sciences.

[35]  Maher Moakher,et al.  The Riemannian Geometry of the Space of Positive-Definite Matrices and Its Application to the Regularization of Positive-Definite Matrix-Valued Data , 2011, Journal of Mathematical Imaging and Vision.

[36]  C. Paige Computational variants of the Lanczos method for the eigenproblem , 1972 .

[37]  Bart G. van Bloemen Waanders,et al.  Fast Algorithms for Bayesian Uncertainty Quantification in Large-Scale Linear Inverse Problems Based on Low-Rank Partial Hessian Approximations , 2011, SIAM J. Sci. Comput..

[38]  Wei Li,et al.  Efficient geostatistical inverse methods for structured and unstructured grids , 2006 .

[39]  A. Abdel-azim Fundamentals of Heat and Mass Transfer , 2011 .

[40]  Michael A. Saunders,et al.  LSRN: A Parallel Iterative Solver for Strongly Over- or Underdetermined Systems , 2011, SIAM J. Sci. Comput..

[41]  Yousef Saad,et al.  Iterative methods for sparse linear systems , 2003 .

[42]  H. Rue,et al.  An explicit link between Gaussian fields and Gaussian Markov random fields: the stochastic partial differential equation approach , 2011 .

[43]  Tiangang Cui,et al.  Dimension-independent likelihood-informed MCMC , 2014, J. Comput. Phys..

[44]  C. Lanczos An iteration method for the solution of the eigenvalue problem of linear differential and integral operators , 1950 .

[45]  Colin Fox,et al.  Posterior Exploration for Computationally Intensive Forward Models , 2011 .

[46]  Ute Beyer Mechanical Behaviour Of Materials At High Temperature , 2016 .

[47]  A. Stuart,et al.  The Bayesian Approach to Inverse Problems , 2013, 1302.6989.

[48]  P. Hansen Regularization,GSVD and truncatedGSVD , 1989 .

[49]  Michael A. Saunders,et al.  LSMR: An Iterative Algorithm for Sparse Least-Squares Problems , 2010, SIAM J. Sci. Comput..

[50]  L. Pardo Statistical Inference Based on Divergence Measures , 2005 .

[51]  Tiangang Cui,et al.  Optimal Low-rank Approximations of Bayesian Linear Inverse Problems , 2014, SIAM J. Sci. Comput..

[52]  P. Absil,et al.  A truncated-CG style method for symmetric generalized eigenvalue problems , 2006 .

[53]  Pierre-Antoine Absil,et al.  Adaptive Model Trust Region Methods for Generalized Eigenvalue Problems , 2005, International Conference on Computational Science.

[54]  Georg Stadler,et al.  Scalable and efficient algorithms for the propagation of uncertainty from data through inference to prediction for large-scale problems, with application to flow of the Antarctic ice sheet , 2014, J. Comput. Phys..

[55]  Gene H. Golub,et al.  An Inverse Free Preconditioned Krylov Subspace Method for Symmetric Generalized Eigenvalue Problems , 2002, SIAM J. Sci. Comput..

[56]  R. Bhatia Positive Definite Matrices , 2007 .

[57]  Pierre-Antoine Absil,et al.  An Implicit Riemannian Trust-Region Method for the Symmetric Generalized Eigenproblem , 2006, International Conference on Computational Science.

[58]  David Higdon,et al.  Adaptive Hessian-Based Nonstationary Gaussian Process Response Surface Method for Probability Density Approximation with Application to Bayesian Solution of Large-Scale Inverse Problems , 2012, SIAM J. Sci. Comput..

[59]  Xavier Pennec,et al.  A Riemannian Framework for Tensor Computing , 2005, International Journal of Computer Vision.

[60]  Masashi Sugiyama,et al.  Geometry-aware principal component analysis for symmetric positive definite matrices , 2017, Machine Learning.

[61]  W. Förstner,et al.  A Metric for Covariance Matrices , 2003 .

[62]  Christoph Schwab,et al.  Sparse, adaptive Smolyak quadratures for Bayesian inverse problems , 2013 .

[63]  David E. Keyes,et al.  Parallel Algorithms for PDE-Constrained Optimization , 2006, Parallel Processing for Scientific Computing.

[64]  M. Saunders,et al.  Towards a Generalized Singular Value Decomposition , 1981 .

[65]  Julianne Chung,et al.  An efficient approach for computing optimal low-rank regularized inverse matrices , 2014, ArXiv.

[66]  Calyampudi R. Rao On the distance between two populations , 1949 .

[67]  N. S. Barnett,et al.  Private communication , 1969 .

[68]  Søren Hauberg,et al.  Manifold Valued Statistics, Exact Principal Geodesic Analysis and the Effect of Linear Approximations , 2010, ECCV.

[69]  Alan S. Willsky,et al.  A Krylov Subspace Method for Covariance Approximation and Simulation of Random Processes and Fields , 2003, Multidimens. Syst. Signal Process..

[70]  P. Mahalanobis On the generalized distance in statistics , 1936 .

[71]  Jörg Liesen,et al.  Convergence analysis of Krylov subspace methods , 2004 .

[72]  Karen Willcox,et al.  Goal-Oriented Inference: Approach, Linear Theory, and Application to Advection Diffusion , 2012, SIAM J. Sci. Comput..

[73]  P. Thomas Fletcher,et al.  Principal geodesic analysis for the study of nonlinear statistics of shape , 2004, IEEE Transactions on Medical Imaging.

[74]  C. Atkinson Rao's distance measure , 1981 .

[75]  Silvere Bonnabel,et al.  Riemannian Metric and Geometric Mean for Positive Semidefinite Matrices of Fixed Rank , 2008, SIAM J. Matrix Anal. Appl..

[76]  A. Quarteroni,et al.  Numerical Approximation of Partial Differential Equations , 2008 .

[77]  Georg Stadler,et al.  Extreme-scale UQ for Bayesian inverse problems governed by PDEs , 2012, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis.

[78]  Anatoli Torokhti,et al.  Generalized Rank-Constrained Matrix Approximations , 2007, SIAM J. Matrix Anal. Appl..

[79]  A. Wood,et al.  Simulation of Stationary Gaussian Processes in [0, 1] d , 1994 .

[80]  Ahmed H. Sameh,et al.  Trace Minimization Algorithm for the Generalized Eigenvalue Problem , 1982, PPSC.

[81]  Dianne P. O'Leary,et al.  Optimal regularized low rank inverse approximation , 2015 .

[82]  Omar Ghattas,et al.  Analysis of the Hessian for inverse scattering problems: I. Inverse shape scattering of acoustic waves , 2012 .

[83]  D. Calvetti,et al.  Priorconditioners for linear systems , 2005 .

[84]  Aaas News,et al.  Book Reviews , 1893, Buffalo Medical and Surgical Journal.

[85]  J. Cullum,et al.  A block Lanczos algorithm for computing the q algebraically largest eigenvalues and a corresponding eigenspace of large, sparse, real symmetric matrices , 1974, CDC 1974.

[86]  Lothar Reichel,et al.  Simplified GSVD computations for the solution of linear discrete ill-posed problems , 2014, J. Comput. Appl. Math..

[87]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[88]  Josef Dick,et al.  Higher order Quasi-Monte Carlo integration for Bayesian Estimation , 2016, 1602.07363.

[89]  Andrew M. Stuart,et al.  Inverse problems: A Bayesian perspective , 2010, Acta Numerica.

[90]  Tiangang Cui,et al.  Likelihood-informed dimension reduction for nonlinear inverse problems , 2014, 1403.4680.

[91]  R. Fisher,et al.  On the Mathematical Foundations of Theoretical Statistics , 1922 .

[92]  Michael A. Saunders,et al.  LSQR: An Algorithm for Sparse Linear Equations and Sparse Least Squares , 1982, TOMS.

[93]  C. Loan Generalizing the Singular Value Decomposition , 1976 .

[94]  L. Skovgaard A Riemannian geometry of the multivariate normal model , 1984 .

[95]  Habib N. Najm,et al.  Dimensionality reduction and polynomial chaos acceleration of Bayesian inference in inverse problems , 2008, J. Comput. Phys..