Neural Spatio-Temporal Point Processes

We propose a new class of parameterizations for spatio-temporal point processes which leverage Neural ODEs as a computational method and enable flexible, high-fidelity models of discrete events that are localized in continuous time and space. Central to our approach is a combination of recurrent continuous-time neural networks with two novel neural architectures, i.e., Jump and Attentive Continuous-time Normalizing Flows. This approach allows us to learn complex distributions for both the spatial and temporal domain and to condition non-trivially on the observed event history. We validate our models on data sets from a wide variety of contexts such as seismology, epidemiology, urban mobility, and neuroscience.

[1]  Eric Jones,et al.  SciPy: Open Source Scientific Tools for Python , 2001 .

[2]  Myle Ott,et al.  fairseq: A Fast, Extensible Toolkit for Sequence Modeling , 2019, NAACL.

[3]  David van Dijk,et al.  TrajectoryNet: A Dynamic Optimal Transport Network for Modeling Cellular Dynamics , 2020, ICML.

[4]  T. Ozaki.  Maximum likelihood estimation of Hawkes' self-exciting point processes , 1979 .

[5]  Aaron C. Courville,et al.  Solving ODE with Universal Flows: Approximation Theory for Flow-Based Models , 2020, ICLR 2020.

[6]  Adam M. Oberman,et al.  How to Train Your Neural ODE: the World of Jacobian and Kinetic Regularization , 2020, ICML.

[7]  Geoffrey E. Hinton,et al.  Layer Normalization , 2016, ArXiv.

[8]  Alexandre Lacoste,et al.  Neural Autoregressive Flows , 2018, ICML.

[9]  Kamalika Chaudhuri,et al.  The Expressive Power of a Class of Normalizing Flow Models , 2020, AISTATS.

[10]  Gaël Varoquaux,et al.  The NumPy Array: A Structure for Efficient Numerical Computation , 2011, Computing in Science & Engineering.

[11]  Austin R. Benson,et al.  Neural Jump Stochastic Differential Equations , 2019, NeurIPS.

[12]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[13]  Guido Rossum,et al.  Python Reference Manual , 2000 .

[14]  Utkarsh Upadhyay,et al.  Recurrent Marked Temporal Point Processes: Embedding Event History to Vector , 2016, KDD.

[15]  Jason Eisner,et al.  The Neural Hawkes Process: A Neurally Self-Modulating Multivariate Point Process , 2016, NIPS.

[16]  Masashi Sugiyama,et al.  Coupling-based Invertible Neural Networks Are Universal Diffeomorphism Approximators , 2020, NeurIPS.

[17]  Emile Mathieu,et al.  Riemannian Continuous Normalizing Flows , 2020, NeurIPS.

[18]  Michael Höhle,et al.  A Space–Time Conditional Intensity Model for Invasive Meningococcal Disease Occurrence , 2012, Biometrics.

[19]  Adrian Baddeley,et al.  Spatial Point Processes and their Applications , 2007 .

[20]  Jure Leskovec,et al.  SEISMIC: A Self-Exciting Point Process Model for Predicting Tweet Popularity , 2015, KDD.

[21]  Yang Li,et al.  Exchangeable Neural ODE for Set Modeling , 2020, NeurIPS.

[22]  J. Møller,et al.  Statistical Inference and Simulation for Spatial Point Processes , 2003 .

[23]  Le Song,et al.  Fake News Mitigation via Point Process Based Intervention , 2017, ICML.

[24]  Adam M. Oberman,et al.  How to train your neural ODE , 2020, ICML.

[25]  David Duvenaud,et al.  Latent ODEs for Irregularly-Sampled Time Series , 2019, ArXiv.

[26]  Marc G. Genton,et al.  Modeling spatio-temporal wildfire ignition point patterns , 2009, Environmental and Ecological Statistics.

[27]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[28]  Pablo Balenzuela,et al.  Criticality in Large-Scale Brain fMRI Dynamics Unveiled by a Novel Point Process Analysis , 2012, Front. Physio..

[29]  Ulrike Goldschmidt,et al.  An Introduction To The Theory Of Point Processes , 2016 .

[30]  Jiawei He,et al.  Point Process Flows , 2019, ArXiv.

[31]  Travis E. Oliphant,et al.  Python for Scientific Computing , 2007, Computing in Science & Engineering.

[32]  J. Skilling.  The Eigenvalues of Mega-dimensional Matrices , 1989 .

[33]  Quoc V. Le,et al.  Searching for Activation Functions , 2018, arXiv.

[34]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[35]  Alex Reinhart,et al.  A Review of Self-Exciting Spatio-Temporal Point Processes and Their Applications , 2017, Statistical Science.

[36]  David Duvenaud,et al.  Neural Ordinary Differential Equations , 2018, NeurIPS.

[37]  John D. Hunter,et al.  Matplotlib: A 2D Graphics Environment , 2007, Computing in Science & Engineering.

[38]  Y. Ogata.  Space-Time Point-Process Models for Earthquake Occurrences , 1998 .

[39]  Yiping Ke,et al.  Tweedie-Hawkes Processes: Interpreting the Phenomena of Outbreaks , 2020, AAAI.

[40]  M. Hutchinson.  A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines , 1989 .

[41]  Wes McKinney,et al.  Python for Data Analysis , 2012 .

[42]  Will Grathwohl.  Scalable Reversible Generative Models with Free-form Continuous Dynamics , 2018 .

[43]  Marc Hoffmann,et al.  A recursive point process model for infectious diseases , 2017, Annals of the Institute of Statistical Mathematics.

[44]  Stefan Behnel,et al.  Cython: The Best of Both Worlds , 2011, Computing in Science & Engineering.

[45]  Andriy Mnih,et al.  The Lipschitz Constant of Self-Attention , 2020, ArXiv.

[46]  Yan Liu,et al.  Recurrent Neural Networks for Multivariate Time Series with Missing Values , 2016, Scientific Reports.

[47]  Scott W. Linderman,et al.  Discovering Latent Network Structure in Point Process Data , 2014, ICML.

[48]  Matthew Le,et al.  Learning Multivariate Hawkes Processes at Scale , 2020, ArXiv.

[49]  Shakir Mohamed,et al.  Variational Inference with Normalizing Flows , 2015, ICML.

[50]  Yoshua Bengio,et al.  NICE: Non-linear Independent Components Estimation , 2014, ICLR.

[51]  Travis E. Oliphant,et al.  Guide to NumPy , 2015 .

[52]  Patrick Forr'e,et al.  Neural Ordinary Differential Equations on Manifolds , 2020, ArXiv.

[53]  et al.,et al.  Jupyter Notebooks - a publishing format for reproducible computational workflows , 2016, ELPUB.

[54]  Prafulla Dhariwal,et al.  Glow: Generative Flow with Invertible 1x1 Convolutions , 2018, NeurIPS.

[55]  Edward De Brouwer,et al.  GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series , 2019, NeurIPS.

[56]  Samy Bengio,et al.  Density estimation using Real NVP , 2016, ICLR.

[57]  Quoc V. Le,et al.  Swish: a Self-Gated Activation Function , 2017, 1710.05941.

[58]  Christopher De Sa,et al.  Neural Manifold Ordinary Differential Equations , 2020, NeurIPS.

[59]  Frederic Paik Schoenberg,et al.  Application of Branching Models in the Study of Invasive Species , 2012 .

[60]  Abhinav Gupta,et al.  BOLD5000, a public fMRI dataset while viewing 5000 visual images , 2018, Scientific Data.

[61]  Jeffrey D. Scargle,et al.  An Introduction to the Theory of Point Processes, Vol. I: Elementary Theory and Methods , 2004, Technometrics.

[62]  Junhyung Park,et al.  A non-parametric Hawkes model of the spread of Ebola in west Africa , 2020, Journal of applied statistics.

[63]  Enzo Tagliazucchi,et al.  The Voxel-Wise Functional Connectome Can Be Efficiently Derived from Co-activations in a Sparse Spatio-Temporal Point-Process , 2016, Front. Neurosci..

[64]  Stephan Günnemann,et al.  Intensity-Free Learning of Temporal Point Processes , 2020, ICLR.

[65]  Greg Mori,et al.  Modeling Continuous Stochastic Processes with Dynamic Normalizing Flows , 2020, NeurIPS.

[66]  Ming-Yu Liu,et al.  PointFlow: 3D Point Cloud Generation With Continuous Normalizing Flows , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[67]  Hongyuan Zha,et al.  Learning Parametric Models for Social Infectivity in Multi-Dimensional Hawkes Processes , 2014, AAAI.

[68]  Leonidas J. Guibas,et al.  CaSPR: Learning Canonical Spatiotemporal Point Cloud Representations , 2020, NeurIPS.

[69]  Yosihiko Ogata,et al.  Statistical Models for Earthquake Occurrences and Residual Analysis for Point Processes , 1988 .