Numerical Gaussian Processes for Time-Dependent and Nonlinear Partial Differential Equations

We introduce the concept of numerical Gaussian processes, which we define as Gaussian processes with covariance functions resulting from temporal discretization of time-dependent partial differential equations. Numerical Gaussian processes, by construction, are designed to deal with cases where (a) all we observe are noisy data on black-box initial conditions, and (b) we are interested in quantifying the uncertainty associated with such noisy data in our solutions to time-dependent partial differential equations. Our method circumvents the need for spatial discretization of the differential operators by proper placement of Gaussian process priors. This is an attempt to construct structured and data-efficient learning machines, which are explicitly informed by the underlying physics that possibly generated the observed data. The effectiveness of the proposed approach is demonstrated through several benchmark problems involving linear and nonlinear time-dependent operators. In all examples, we are able to r...

[1]  C. Runge.  Ueber die numerische Auflösung von Differentialgleichungen , 1895 .

[2]  H. Poincaré.  Calcul des Probabilités , 1912 .

[3]  N. Aronszajn.  Theory of Reproducing Kernels. , 1950 .

[4]  A Tikhonov,et al.  Solution of Incorrectly Formulated Problems and the Regularization Method , 1963 .

[5]  R. Alexander.  Diagonally implicit runge-kutta methods for stiff odes , 1977 .

[6]  A. N. Tikhonov,et al.  Solutions of ill-posed problems , 1977 .

[7]  Temple F. Smith.  Occam's razor , 1980, Nature.

[8]  C. Basdevant,et al.  Spectral and finite difference solutions of the Burgers equation , 1986 .

[9]  S. Gupta,et al.  Statistical decision theory and related topics IV , 1988 .

[10]  Saburou Saitoh,et al.  Theory of Reproducing Kernels and Its Applications , 1988 .

[11]  F. Girosi,et al.  Networks for approximation and learning , 1990, Proc. IEEE.

[12]  J. Lambert.  Numerical Methods for Ordinary Differential Equations , 1991 .

[13]  Geoffrey E. Hinton,et al.  Bayesian Learning for Neural Networks , 1995 .

[14]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[15]  G. Akrivis.  A First Course In The Numerical Analysis Of Differential Equations [Book News & Reviews] , 1998, IEEE Computational Science and Engineering.

[16]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[17]  Klaus Ritter,et al.  Bayesian numerical analysis , 2000 .

[18]  George Eastman House,et al.  Sparse Bayesian Learning and the Relevan e Ve tor Ma hine , 2001 .

[19]  Bernhard Schölkopf,et al.  Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond , 2005, IEEE Transactions on Neural Networks.

[20]  Anthony Widjaja,et al.  Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond , 2003, IEEE Transactions on Neural Networks.

[21]  A. Berlinet,et al.  Reproducing kernel Hilbert spaces in probability and statistics , 2004 .

[22]  Zoubin Ghahramani,et al.  Sparse Gaussian Processes using Pseudo-inputs , 2005, NIPS.

[23]  John C. Adams,et al.  An Attempt to Test the Theories of Capillary Action: By Comparing the Theoretical and Measured Forms of Drops of Fluid , 2007 .

[24]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[25]  Jouni Hartikainen,et al.  Kalman filtering and smoothing solutions to temporal Gaussian process regression models , 2010, 2010 IEEE International Workshop on Machine Learning for Signal Processing.

[26]  Kevin P. Murphy,et al.  Machine learning - a probabilistic perspective , 2012, Adaptive computation and machine learning series.

[27]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[28]  C. Scovel,et al.  Brittleness of Bayesian Inference Under Finite Information in a Continuous World , 2013, 1304.6772.

[29]  Neil D. Lawrence,et al.  Gaussian Processes for Big Data , 2013, UAI.

[30]  David Duvenaud,et al.  Probabilistic ODE Solvers with Runge-Kutta Means , 2014, NIPS.

[31]  Michael A. Osborne,et al.  Probabilistic numerics and uncertainty in computations , 2015, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[32]  Zoubin Ghahramani,et al.  Probabilistic machine learning and artificial intelligence , 2015, Nature.

[33]  Michael I. Jordan,et al.  Machine learning: Trends, perspectives, and prospects , 2015, Science.

[34]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[35]  Paris Perdikaris,et al.  Inferring solutions of differential equations using noisy multi-fidelity data , 2016, J. Comput. Phys..

[36]  Andrew M. Stuart,et al.  Statistical analysis of differential equations: introducing probability measures on numerical solutions , 2016, Statistics and Computing.

[37]  Paris Perdikaris,et al.  Machine learning of linear differential equations using Gaussian processes , 2017, J. Comput. Phys..

[38]  Andrew M. Stuart,et al.  Posterior consistency for Gaussian process approximations of Bayesian posterior distributions , 2016, Math. Comput..

[39]  Maziar Raissi,et al.  Parametric Gaussian process regression for big data , 2017, Computational Mechanics.