Constraining the Dynamics of Deep Probabilistic Models

We introduce a novel generative formulation of deep probabilistic models implementing "soft" constraints on the dynamics of the functions they can model. In particular we develop a flexible methodological framework where the modeled functions and derivatives of a given order are subject to inequality or equality constraints. We characterize the posterior distribution over model and constraint parameters through stochastic variational inference techniques. As a result, the proposed approach allows for accurate and scalable uncertainty quantification of predictions and parameters. We demonstrate the application of equality constraints in the challenging problem of parameter inference in ordinary differential equation models, while we showcase the application of inequality constraints on monotonic regression on count data. The proposed approach is extensively tested in several experimental settings, leading to highly competitive results in challenging modeling applications, while offering high expressiveness, flexibility and scalability.

[1]  James D. Broffiti.  INCREASING AND INCREASING CONVEX BAYESIAN GRADUATION , 1988 .

[2]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[3]  Carl E. Rasmussen,et al.  In Advances in Neural Information Processing Systems , 2011 .

[4]  Maurizio Filippone,et al.  Random Feature Expansions for Deep Gaussian Processes , 2016, ICML.

[5]  Neil D. Lawrence,et al.  Linear Latent Force Models Using Gaussian Processes , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Dirk Husmeier,et al.  Gradient Matching Methods for Computational Inference in Mechanistic Models for Systems Biology: A Review and Comparative Analysis , 2015, Front. Bioeng. Biotechnol..

[7]  Benjamin Recht,et al.  Random Features for Large-Scale Kernel Machines , 2007, NIPS.

[8]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[9]  Neil D. Lawrence,et al.  Accelerating Bayesian Inference over Nonlinear Differential Equations with Gaussian Processes , 2008, NIPS.

[10]  Dirk Husmeier,et al.  Fast Parameter Inference in Nonlinear Dynamical Systems using Iterative Gradient Matching , 2016, ICML.

[11]  Zoubin Ghahramani,et al.  Probabilistic machine learning and artificial intelligence , 2015, Nature.

[12]  Hervé Delingette,et al.  Efficient probabilistic model personalization integrating uncertainty on data and parameters: Application to eikonal-diffusion models in cardiac electrophysiology. , 2011, Progress in biophysics and molecular biology.

[13]  Hélène Jacqmin-Gadda,et al.  Estimating long-term multivariate progression from short-term data , 2014, Alzheimer's & Dementia.

[14]  Jiguo Cao,et al.  Parameter estimation for differential equations: a generalized smoothing approach , 2007 .

[15]  Tom Minka,et al.  Expectation Propagation for approximate Bayesian inference , 2001, UAI.

[16]  Hulin Wu,et al.  Parameter Estimation for Differential Equation Models Using a Framework of Measurement Error in Regression Models , 2008, Journal of the American Statistical Association.

[17]  Sébastien Da Veiga,et al.  Gaussian process modeling with inequality constraints , 2012 .

[18]  Stefan Bauer,et al.  Scalable Variational Inference for Dynamical Systems , 2017, NIPS.

[19]  Elliott W. Montroll,et al.  Nonlinear Population Dynamics. (Book Reviews: On the Volterra and Other Nonlinear Models of Interacting Populations) , 1971 .

[20]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[21]  David Campbell,et al.  Smooth functional tempering for nonlinear differential equation models , 2012, Stat. Comput..

[22]  Raquel Urtasun,et al.  Implicitly Constrained Gaussian Process Regression for Monocular Non-Rigid Pose Estimation , 2010, NIPS.

[23]  Maurizio Filippone,et al.  Probabilistic disease progression modeling to characterize diagnostic uncertainty: Application to staging and prediction in Alzheimer's disease , 2017, NeuroImage.

[24]  Aki Vehtari,et al.  GPstuff: Bayesian modeling with Gaussian processes , 2013, J. Mach. Learn. Res..

[25]  Dan Cornford,et al.  Variational mean-field algorithm for efficient inference in large systems of stochastic differential equations. , 2015, Physical review. E, Statistical, nonlinear, and soft matter physics.

[26]  David B. Dunson,et al.  Mechanistic Hierarchical Gaussian Processes , 2014, Journal of the American Statistical Association.

[27]  R. FitzHugh.  Mathematical models of threshold phenomena in the nerve membrane , 1955 .

[28]  Dirk Husmeier,et al.  Statistical inference in mechanistic models: time warping for improved gradient matching , 2018, Comput. Stat..

[29]  Carl M. O’Brien,et al.  Nonparametric Estimation under Shape Constraints: Estimators, Algorithms and Asymptotics , 2016 .

[30]  Aki Vehtari,et al.  Gaussian processes with monotonicity information , 2010, AISTATS.

[31]  David Duvenaud,et al.  Probabilistic ODE Solvers with Runge-Kutta Means , 2014, NIPS.

[32]  Ernst Wit,et al.  Reproducing kernel Hilbert space based estimation of systems of ordinary differential equations , 2013, Pattern Recognit. Lett..

[33]  Dirk Husmeier,et al.  ODE parameter inference using adaptive gradient matching with Gaussian processes , 2013, AISTATS.

[34]  David Barber,et al.  Gaussian Processes for Bayesian Estimation in Ordinary Differential Equations , 2014, ICML.

[35]  J. Varah.  A Spline Least Squares Method for Numerical Parameter Estimation in Differential Equations , 1982 .

[36]  Neil D. Lawrence,et al.  Deep Gaussian Processes , 2012, AISTATS.

[37]  Mary C. Meyer.  INFERENCE USING SHAPE-RESTRICTED REGRESSION SPLINES , 2008, 0811.1705.

[38]  K. Emanuel,et al.  Optimal Sites for Supplementary Weather Observations: Simulation with a Small Model , 1998 .

[39]  Geurt Jongbloed,et al.  Nonparametric Estimation under Shape Constraints , 2014 .

[40]  Dominique Bonvin,et al.  Shape constrained splines as transparent black-box models for bioprocess modeling , 2017, Comput. Chem. Eng..

[41]  Carl E. Rasmussen,et al.  Manifold Gaussian Processes for regression , 2014, 2016 International Joint Conference on Neural Networks (IJCNN).

[42]  Mark A. Girolami,et al.  Bayesian ranking of biochemical system models , 2008, Bioinform..

[43]  Dirk Husmeier,et al.  Controversy in mechanistic modelling with Gaussian processes , 2015, ICML.