Relational VAE: A Continuous Latent Variable Model for Graph Structured Data

Graph Networks (GNs) enable the fusion of prior knowledge and relational reasoning with flexible function approximations. In this work, a general GN-based model is proposed which takes full advantage of the relational modeling capabilities of GNs and extends these to probabilistic modeling with Variational Bayes (VB). To that end, we combine complementary pre-existing approaches on VB for graph data and propose an approach that relies on graph-structured latent and conditioning variables. It is demonstrated that Neural Processes can also be viewed through the lens of the proposed model. We show applications on the problem of structured probability density modeling for simulated and real wind farm monitoring data, as well as on the meta-learning of simulated Gaussian Process data. We release the source code, along with the simulated datasets.

[1]  Max Welling,et al.  E(n) Equivariant Graph Neural Networks , 2021, ICML.

[2]  Shu-Tao Xia,et al.  Stochastic Deep Gaussian Processes over Graphs , 2020, NeurIPS.

[3]  Aviral Kumar,et al.  Graph Normalizing Flows , 2019, NeurIPS.

[4]  Nikos Komodakis,et al.  GraphVAE: Towards Generation of Small Graphs Using Variational Autoencoders , 2018, ICANN.

[5]  R. Zemel,et al.  Neural Relational Inference for Interacting Systems , 2018, ICML.

[6]  Alexander J. Smola,et al.  Deep Sets , 2017, 1703.06114.

[7]  Pietro Liò,et al.  Principal Neighbourhood Aggregation for Graph Nets , 2020, NeurIPS.

[8]  Daniel T. Kaffine,et al.  Costs and consequences of wind turbine wake effects arising from uncoordinated wind energy development , 2018, Nature Energy.

[9]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[10]  Wei Chu,et al.  Relational Learning with Gaussian Processes , 2006, NIPS.

[11]  Christopher Burgess,et al.  beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework , 2016, ICLR 2016.

[12]  Shakir Mohamed,et al.  Variational Inference with Normalizing Flows , 2015, ICML.

[13]  Samy Bengio,et al.  Density estimation using Real NVP , 2016, ICLR.

[14]  Klamer Schutte,et al.  The Functional Neural Process , 2019, NeurIPS.

[15]  Pierre Baldi,et al.  Continuous Representation of Molecules using Graph Variational Autoencoder , 2020, AAAI Spring Symposium: MLPS.

[16]  Xiaodong Liu,et al.  Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing , 2019, NAACL.

[17]  James Bleeg,et al.  A Graph Neural Network Surrogate Model for the Prediction of Turbine Interaction Loss , 2020, Journal of Physics: Conference Series.

[18]  Jinkyoo Park,et al.  Physics-induced graph neural network: An application to wind-farm power estimation , 2019, Energy.

[19]  Meire Fortunato,et al.  Learning Mesh-Based Simulation with Graph Networks , 2020, ArXiv.

[20]  Samuel S. Schoenholz,et al.  Neural Message Passing for Quantum Chemistry , 2017, ICML.

[21]  Daan Wierstra,et al.  Stochastic Backpropagation and Approximate Inference in Deep Generative Models , 2014, ICML.

[22]  Qi Wang,et al.  Doubly Stochastic Variational Inference for Neural Processes with Hierarchical Latent Variables , 2020, ICML.

[23]  Yee Whye Teh,et al.  Attentive Neural Processes , 2019, ICLR.

[24]  Stefano Ermon,et al.  Graphite: Iterative Generative Modeling of Graphs , 2018, ICML.

[25]  Yee Whye Teh,et al.  Neural Processes , 2018, ArXiv.

[26]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[27]  Dmitry Vetrov,et al.  Variational Autoencoder with Arbitrary Conditioning , 2018, ICLR.

[28]  Prateek Mittal,et al.  Graph Data Anonymization, De-Anonymization Attacks, and De-Anonymizability Quantification: A Survey , 2017, IEEE Communications Surveys & Tutorials.

[29]  Yee Whye Teh,et al.  Conditional Neural Processes , 2018, ICML.

[30]  Ben Poole,et al.  Categorical Reparameterization with Gumbel-Softmax , 2016, ICLR.

[31]  Alexander A. Alemi,et al.  Fixing a Broken ELBO , 2017, ICML.

[32]  Samy Bengio,et al.  Generating Sentences from a Continuous Space , 2015, CoNLL.

[33]  Ole Winther,et al.  Ladder Variational Autoencoders , 2016, NIPS.

[34]  Pietro Lio,et al.  Message Passing Neural Processes , 2020, ArXiv.

[35]  Sandia Report,et al.  Definition of a 5MW/61.5m Wind Turbine Blade Reference Model , 2013 .

[36]  Richard E. Turner,et al.  Convolutional Conditional Neural Processes , 2019, ICLR.

[37]  Yee Whye Teh,et al.  The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables , 2016, ICLR.

[38]  Sanjiva K. Lele,et al.  Wind farm power optimization through wake steering , 2019, Proceedings of the National Academy of Sciences.

[39]  Yin Cheng Ng,et al.  Bayesian Semi-supervised Learning with Graph Gaussian Processes , 2018, NeurIPS.

[40]  Max Welling,et al.  Variational Graph Auto-Encoders , 2016, ArXiv.

[41]  Jure Leskovec,et al.  Learning to Simulate Complex Physics with Graph Networks , 2020, ICML.