Neural Jump Stochastic Differential Equations

Many time series are effectively generated by a combination of deterministic continuous flows along with discrete jumps sparked by stochastic events. However, we usually do not have the equation of motion describing the flows, or how they are affected by jumps. To this end, we introduce Neural Jump Stochastic Differential Equations that provide a data-driven approach to learn continuous and discrete dynamic behavior, i.e., hybrid systems that both flow and jump. Our approach extends the framework of Neural Ordinary Differential Equations with a stochastic process term that models discrete events. We then model temporal point processes with a piecewise-continuous latent trajectory, where the discontinuities are caused by stochastic events whose conditional intensity depends on the latent state. We demonstrate the predictive capabilities of our model on a range of synthetic and real-world marked point process datasets, including classical point processes (such as Hawkes processes), awards on Stack Overflow, medical records, and earthquake monitoring.

[1]  Isabel Valera,et al.  Modeling Adoption and Usage of Competing Products , 2014, 2015 IEEE International Conference on Data Mining.

[2]  Yosihiko Ogata,et al.  Inference for earthquake models: A self-correcting model , 1984 .

[3]  Shuang Li,et al.  COEVOLVE: A Joint Point Process Model for Information Diffusion and Network Co-evolution , 2015, NIPS.

[4]  Hamid R. Rabiee,et al.  RedQueen: An Online Algorithm for Smart Broadcasting in Social Networks , 2016, WSDM.

[5]  John Lygeros,et al.  A Stochastic Hybrid Model for Air Traffic Control Simulation , 2004, HSCC.

[6]  Lexing Ying,et al.  A Multiscale Neural Network Based on Hierarchical Matrices , 2018, Multiscale Model. Simul..

[7]  Scott Sanner,et al.  Expecting to be HIP: Hawkes Intensity Processes for Social Media Popularity , 2016, WWW.

[8]  Michael S. Branicky,et al.  Introduction to Hybrid Systems , 2005, Handbook of Networked and Embedded Control Systems.

[9]  Vaibhav Dixit,et al.  A Comparison of Automatic Differentiation and Continuous Sensitivity Analysis for Derivatives of Differential Equation Solutions , 2018, 2021 IEEE High Performance Extreme Computing Conference (HPEC).

[10]  George E. Karniadakis,et al.  Hidden physics models: Machine learning of nonlinear partial differential equations , 2017, J. Comput. Phys..

[11]  Maxim Raginsky,et al.  Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit , 2019, ArXiv.

[12]  Adrian Sandu,et al.  Adjoint sensitivity analysis of hybrid multibody dynamical systems , 2018, Multibody System Dynamics.

[13]  Alan Edelman,et al.  A Differentiable Programming System to Bridge Machine Learning and Scientific Computing , 2019, ArXiv.

[14]  Katherine A. Heller,et al.  Modelling Reciprocating Relationships with Hawkes Processes , 2012, NIPS.

[15]  S. Ross,et al.  The valuation of options for alternative stochastic processes , 1976 .

[16]  Y. Ogata.  Seismicity Analysis through Point-process Modeling: A Review , 1999 .

[17]  João Pedro Hespanha,et al.  Stochastic Hybrid Systems: Application to Communication Networks , 2004, HSCC.

[18]  David Duvenaud,et al.  Neural Ordinary Differential Equations , 2018, NeurIPS.

[19]  Hongyuan Zha,et al.  Learning Hawkes Processes from Short Doubly-Censored Event Sequences , 2017, ICML.

[20]  Utkarsh Upadhyay,et al.  Recurrent Marked Temporal Point Processes: Embedding Event History to Vector , 2016, KDD.

[21]  Stefano Peluchetti,et al.  Infinitely deep neural networks as diffusion processes , 2020, AISTATS.

[22]  Le Song,et al.  A Stochastic Differential Equation Framework for Guiding Online User Activities in Closed Loop , 2016, AISTATS.

[23]  Hongyuan Zha,et al.  Modeling the Intensity Function of Point Process Via Recurrent Neural Networks , 2017, AAAI.

[24]  Jason Eisner,et al.  The Neural Hawkes Process: A Neurally Self-Modulating Multivariate Point Process , 2016, NIPS.

[25]  Daryl J. Daley,et al.  An Introduction to the Theory of Point Processes , 2013 .

[26]  Ulrike Goldschmidt,et al.  An Introduction To The Theory Of Point Processes , 2016 .

[27]  R. C. Merton,et al.  Option pricing when underlying stock returns are discontinuous , 1976 .

[28]  Stephen J. Hardiman,et al.  Critical reflexivity in financial markets: a Hawkes process analysis , 2013, 1302.1405.

[29]  Jure Leskovec,et al.  Steering user behavior with badges , 2013, WWW.

[30]  Vaibhav Dixit,et al.  DiffEqFlux.jl - A Julia Library for Neural Differential Equations , 2019, ArXiv.

[31]  Hongyuan Zha,et al.  Learning Parametric Models for Social Infectivity in Multi-Dimensional Hawkes Processes , 2014, AAAI.

[32]  T. Taimre,et al.  Hawkes Processes , 2015, 1507.02822.

[33]  Johannes Schumacher,et al.  An Introduction to Hybrid Dynamical Systems, Springer Lecture Notes in Control and Information Sciences 251 , 1999 .

[34]  Renaud Lambiotte,et al.  TiDeH: Time-Dependent Hawkes Process for Predicting Retweet Dynamics , 2016, ICWSM.

[35]  Le Song,et al.  Shaping Social Activity by Incentivizing Users , 2014, NIPS.

[36]  Bin Dong,et al.  PDE-Net: Learning PDEs from Data , 2017, ICML.

[37]  G. Karniadakis,et al.  Multistep Neural Networks for Data-driven Discovery of Nonlinear Dynamical Systems , 2018, 1801.01236.

[38]  Hongyuan Zha,et al.  A Dirichlet Mixture Model of Hawkes Processes for Event Sequence Clustering , 2017, NIPS.

[39]  David Duvenaud,et al.  Latent ODEs for Irregularly-Sampled Time Series , 2019, ArXiv.

[40]  Lexing Ying,et al.  Solving for high-dimensional committor functions using artificial neural networks , 2018, Research in the Mathematical Sciences.

[41]  Le Song,et al.  Wasserstein Learning of Deep Generative Point Process Models , 2017, NIPS.

[42]  A. Stephen McGough,et al.  Black-Box Variational Inference for Stochastic Differential Equations , 2018, ICML.

[43]  Scott Sanner,et al.  Can this video be promoted? - Endogenous and exogenous popularity processes in social media , 2016, ArXiv.

[44]  Edward R. Dougherty,et al.  Review of stochastic hybrid systems with applications in biological systems modeling and analysis , 2017, EURASIP J. Bioinform. Syst. Biol..

[45]  A. Stomakhin,et al.  Reconstruction of missing data in social networks based on temporal patterns of interactions , 2011 .

[46]  Hongyuan Zha,et al.  Dyadic event attribution in social networks with mixtures of hawkes processes , 2013, CIKM.

[47]  Lexing Ying,et al.  Solving parametric PDE problems with artificial neural networks , 2017, European Journal of Applied Mathematics.

[48]  Le Song,et al.  Learning Triggering Kernels for Multi-dimensional Hawkes Processes , 2013, ICML.