Kullback-Leibler Approximation for Probability Measures on Infinite Dimensional Spaces

In a variety of applications it is important to extract information from a probability measure $\mu$ on an infinite dimensional space. Examples include the Bayesian approach to inverse problems and (possibly conditioned) continuous time Markov processes. It may then be of interest to find a measure $\nu$, from within a simple class of measures, which approximates $\mu$. This problem is studied in the case where the Kullback--Leibler divergence is employed to measure the quality of the approximation. A calculus of variations viewpoint is adopted, and the particular case where $\nu$ is chosen from the set of Gaussian measures is studied in detail. Basic existence and uniqueness theorems are established, together with properties of minimizing sequences. Furthermore, parameterization of the class of Gaussians through the mean and inverse covariance is introduced, the need for regularization is explained, and a regularized minimization is studied in detail. The calculus of variations framework resulting from t...

[1]  A. Stuart,et al.  Signal processing problems on function space: Bayesian formulation, stochastic PDEs and effective MCMC methods , 2011 .

[2]  M. Reed,et al.  Methods of Modern Mathematical Physics. 2. Fourier Analysis, Self-adjointness , 1975 .

[3]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[4]  Markos A. Katsoulakis,et al.  COARSE-GRAINING SCHEMES AND A POSTERIORI ERROR ESTIMATES FOR STOCHASTIC LATTICE SYSTEMS , 2006, math/0608007.

[5]  V. Bogachev.  Gaussian Measures on a , 2022 .

[6]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[7]  A. Üstünel,et al.  Monge-Kantorovitch Measure Transportation and Monge-Ampère Equation on Wiener Space , 2004 .

[8]  Tim Hesterberg,et al.  Monte Carlo Strategies in Scientific Computing , 2002, Technometrics.

[9]  A. M. Stuart,et al.  Γ-Limit for Transition Paths of Maximal Probability , 2011, 1101.3920.

[10]  Andrew M. Stuart,et al.  Uncertainty Quantification and Weak Approximation of an Elliptic Inverse Problem , 2011, SIAM J. Numer. Anal..

[11]  C. Villani.  Optimal Transport: Old and New , 2008 .

[12]  M. Reed.  Methods of Modern Mathematical Physics. I: Functional Analysis , 1972 .

[13]  Petr Plechác,et al.  Information-theoretic tools for parametrized coarse-graining of non-equilibrium extended systems , 2013, The Journal of chemical physics.

[14]  Andrew J Majda,et al.  Improving model fidelity and sensitivity for complex systems through empirical information theory , 2011, Proceedings of the National Academy of Sciences.

[15]  Mark S. C. Reed,et al.  Method of Modern Mathematical Physics , 1972 .

[16]  J. Lynch,et al.  A weak convergence approach to the theory of large deviations , 1997 .

[17]  Gideon Simpson,et al.  Algorithms for Kullback-Leibler Approximation of Probability Measures in Infinite Dimensions , 2014, SIAM J. Sci. Comput..

[18]  L. Ambrosio,et al.  Gradient Flows: In Metric Spaces and in the Space of Probability Measures , 2005 .

[19]  A. Stuart,et al.  MAP estimators and their consistency in Bayesian nonparametric inverse problems , 2013, 1303.4795.

[20]  S. Crawford,et al.  Volume 1 , 2012, Journal of Diabetes Investigation.

[21]  Andrew J. Majda,et al.  Quantifying the predictive skill in long-range forecasting. Part II: Model error in coarse-grained Markov models with application to ocean-circulation regimes , 2012 .

[22]  J. Ball,et al.  A numerical method for detecting singular minimizers , 1987 .

[23]  A. M. Stuart,et al.  Posterior consistency via precision operators for Bayesian nonparametric drift estimation in SDEs , 2012, 1202.0976.

[24]  Kristian Kirsch,et al.  Methods Of Modern Mathematical Physics , 2016 .

[25]  Le Gall,et al.  Mouvement brownien, martingales et calcul stochastique , 2013 .

[26]  Barbara Schneider,et al.  Basel , 2000 .

[27]  Maurizio Dapor.  Monte Carlo Strategies , 2020, Transport of Energetic Electrons in Solids.

[28]  Dan Cornford,et al.  Gaussian Process Approximations of Stochastic Differential Equations , 2007, Gaussian Processes in Practice.

[29]  James C. Robinson,et al.  Bayesian inverse problems for functions and applications to fluid mechanics , 2009 .

[30]  Andrew M. Stuart,et al.  Inverse problems: A Bayesian perspective , 2010, Acta Numerica.

[31]  I. Csiszár.  $I$-Divergence Geometry of Probability Distributions and Minimization Problems , 1975 .

[32]  J. Zabczyk,et al.  Stochastic Equations in Infinite Dimensions , 2008 .

[33]  R. Bass,et al.  Review: P. Billingsley, Convergence of probability measures , 1971 .

[34]  O. H. Lowry.  Academic press. , 1972, Analytical chemistry.

[35]  R. McCann.  A Convexity Principle for Interacting Gases , 1997 .

[36]  E. Somersalo,et al.  Statistical and computational inverse problems , 2004 .

[37]  Martin Hairer,et al.  An Introduction to Stochastic PDEs , 2009, 0907.4178.

[38]  Lea Fleischer,et al.  Regularization of Inverse Problems , 1996 .

[39]  Andrew J. Majda,et al.  Quantifying the Predictive Skill in Long-Range Forecasting. Part I: Coarse-Grained Predictions in a Simple Ocean Model , 2012 .

[40]  Dan Cornford,et al.  Variational Inference for Diffusion Processes , 2007, NIPS.

[41]  Michael I. Jordan,et al.  Learning with Mixtures of Trees , 2001, J. Mach. Learn. Res..