Equivariant Normalizing Flows for Point Processes and Sets

A point process describes how random sets of exchangeable points are generated. The points usually influence the positions of each other via attractive and repulsive forces. To model this behavior, it is enough to transform the samples from the uniform process with a sufficiently complex equivariant function. However, learning the parameters of the resulting process is challenging since the likelihood is hard to estimate and often intractable. This leads us to our proposed model - CONFET. Based on continuous normalizing flows, it allows arbitrary interactions between points while having tractable likelihood. Experiments on various real and synthetic datasets show the improved performance of our new scalable approach.

[1]  Hugo Larochelle,et al.  MADE: Masked Autoencoder for Distribution Estimation , 2015, ICML.

[2]  Iain Murray,et al.  Neural Spline Flows , 2019, NeurIPS.

[3]  Jonathon S. Hare,et al.  FSPool: Learning Set Representations with Featurewise Sort Pooling , 2019, ICLR.

[4]  Yee Whye Teh,et al.  Set Transformer , 2018, ICML.

[5]  Jure Leskovec,et al.  node2vec: Scalable Feature Learning for Networks , 2016, KDD.

[6]  Arthur Gretton,et al.  BRUNO: A Deep Recurrent Model for Exchangeable Data , 2018, NeurIPS.

[7]  Jure Leskovec,et al.  Friendship and mobility: user movement in location-based social networks , 2011, KDD.

[8]  Yang Li,et al.  Exchangeable Neural ODE for Set Modeling , 2020, NeurIPS.

[9]  J. Neyman,et al.  Statistical Approach to Problems of Cosmology , 1958 .

[10]  David Duvenaud,et al.  Learning Differential Equations that are Easy to Solve , 2020, NeurIPS.

[11]  Samy Bengio,et al.  Density estimation using Real NVP , 2016, ICLR.

[12]  B. D. Finetti.  La prévision : ses lois logiques, ses sources subjectives , 1937 .

[13]  Jason Eisner,et al.  The Neural Hawkes Process: A Neurally Self-Modulating Multivariate Point Process , 2016, NIPS.

[14]  Daryl J. Daley,et al.  An Introduction to the Theory of Point Processes , 2013 .

[15]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[17]  M. Hutchinson.  A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines , 1989 .

[18]  David Duvenaud,et al.  FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models , 2018, ICLR.

[19]  Bin Dong,et al.  Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations , 2017, ICML.

[20]  Yue Wang,et al.  PointGrow: Autoregressively Learned Point Cloud Generation with Self-Attention , 2018, 2020 IEEE Winter Conference on Applications of Computer Vision (WACV).

[21]  Andrew Jaegle,et al.  Hamiltonian Generative Networks , 2020, ICLR.

[22]  Jure Leskovec,et al.  GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Models , 2018, ICML.

[23]  David Duvenaud,et al.  Neural Ordinary Differential Equations , 2018, NeurIPS.

[24]  Kurt Keutzer,et al.  ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs , 2019, IJCAI.

[25]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[26]  D. Stoyan,et al.  Stochastic Geometry and Its Applications , 1989 .

[27]  Michael A. Osborne,et al.  On the Limitations of Representing Functions on Sets , 2019, ICML.

[28]  Ruslan Salakhutdinov,et al.  Importance Weighted Autoencoders , 2015, ICLR.

[29]  Iain Murray,et al.  Masked Autoregressive Flow for Density Estimation , 2017, NIPS.

[30]  Chang Zhou,et al.  Variational Autoencoders for Highly Multivariate Spatial Point Processes Intensities , 2020, ICLR.

[31]  Frank Noé,et al.  Equivariant Flows: exact likelihood generative learning for symmetric densities , 2020, ICML.

[32]  Gal Chechik,et al.  On Learning Sets of Symmetric Elements , 2020, ICML.

[33]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Alexander J. Smola,et al.  Deep Sets , 2017, 1703.06114.

[35]  Sekhar Tatikonda,et al.  Adaptive Checkpoint Adjoint Method for Gradient Estimation in Neural ODE , 2020, ICML.

[36]  David Duvenaud,et al.  Neural Networks with Cheap Differential Operators , 2019, NeurIPS.

[37]  J. Besag.  Statistical Analysis of Non-Lattice Data , 1975 .

[38]  Max Welling,et al.  Improved Variational Inference with Inverse Autoregressive Flow , 2016, NIPS 2016.

[39]  Yee Whye Teh,et al.  Augmented Neural ODEs , 2019, NeurIPS.

[40]  Nhan Dam,et al.  Model-based learning for point pattern data , 2017, Pattern Recognit..

[41]  S. M. Ali Eslami,et al.  PolyGen: An Autoregressive Generative Model of 3D Meshes , 2020, ICML.

[42]  Ming-Yu Liu,et al.  PointFlow: 3D Point Cloud Generation With Continuous Normalizing Flows , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[43]  F. Papangelou,et al.  The conditional intensity of general point processes and an application to line processes , 1974 .

[44]  M. Thomas.  A generalization of Poisson's binomial limit for use in ecology. , 1949, Biometrika.

[45]  A. Baddeley,et al.  Practical Maximum Pseudolikelihood for Spatial Point Patterns , 1998, Advances in Applied Probability.

[46]  David A. Ham,et al.  Automated Derivation of the Adjoint of High-Level Transient Finite Element Programs , 2013, SIAM J. Sci. Comput..

[47]  Geoffrey E. Hinton,et al.  Layer Normalization , 2016, ArXiv.

[48]  Yang Li,et al.  Exchangeable Generative Models with Flow Scans , 2019, AAAI.

[49]  G. Casella,et al.  Statistical Inference , 2003, Encyclopedia of Social Network Analysis and Mining.

[50]  F. Huang,et al.  Improvements of the Maximum Pseudo-Likelihood Estimators in Various Spatial Statistical Models , 1999 .

[51]  Adam M. Oberman,et al.  How to train your neural ODE , 2020, ICML.

[52]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[53]  Merlise A. Clyde,et al.  Logistic regression for spatial pair-potential models , 1991 .

[54]  Yoshua Bengio,et al.  NICE: Non-linear Independent Components Estimation , 2014, ICLR.

[55]  Jesper Møller,et al.  An Introduction to Simulation-Based Inference for Spatial Point Processes , 2003 .

[56]  Y. Ogata,et al.  Estimation of Interaction Potentials of Marked Spatial Point Patterns Through the Maximum Likelihood Method , 1985 .

[57]  D. Cox.  Some Statistical Methods Connected with Series of Events , 1955 .

[58]  Ryan L. Murphy,et al.  Janossy Pooling: Learning Deep Permutation-Invariant Functions for Variable-Size Inputs , 2018, ICLR.

[59]  J. Dormand,et al.  A family of embedded Runge-Kutta formulae , 1980 .

[60]  Eric Nalisnick,et al.  Normalizing Flows for Probabilistic Modeling and Inference , 2019, J. Mach. Learn. Res..

[61]  Ben O'Neill,et al.  Exchangeability, Correlation, and Bayes' Effect , 2009 .

[62]  D. Ruelle.  Statistical Mechanics: Rigorous Results , 1999 .