A Systematic Survey on Deep Generative Models for Graph Generation

Graphs are important data representations for describing objects and their relationships, which appear in a wide diversity of real-world scenarios. As one of a critical problem in this area, graph generation considers learning the distributions of given graphs and generating more novel graphs. Owing to its wide range of applications, generative models for graphs have a rich history, which, however, are traditionally hand-crafted and only capable of modeling a few statistical properties of graphs. Recent advances in deep generative models for graph generation is an important step towards improving the fidelity of generated graphs and paves the way for new kinds of applications. This article provides an extensive overview of the literature in the field of deep generative models for graph generation. Firstly, the formal definition of deep generative models for the graph generation as well as preliminary knowledge is provided. Secondly, two taxonomies of deep generative models for unconditional, and conditional graph generation respectively are proposed; the existing works of each are compared and analyzed. After that, an overview of the evaluation metrics in this specific domain is provided. Finally, the applications that deep graph generation enables are summarized and five promising future research directions are highlighted.

[1]  Regina Barzilay,et al.  Learning Multimodal Graph-to-Graph Translation for Molecular Optimization , 2018, ICLR.

[2]  Jing Liu,et al.  A Dynamic Multiagent Genetic Algorithm for Gene Regulatory Network Reconstruction Based on Fuzzy Cognitive Maps , 2016, IEEE Transactions on Fuzzy Systems.

[3]  Iain Murray,et al.  Masked Autoregressive Flow for Density Estimation , 2017, NIPS.

[4]  Sayan Ranu,et al.  GraphGen: A Scalable Approach to Domain-agnostic Labeled Graph Generation , 2020, WWW.

[5]  Wanxiang Che,et al.  A Neural Transition-Based Approach for Semantic Dependency Graph Parsing , 2018, AAAI.

[6]  Peter Dayan,et al.  Q-learning , 1992, Machine Learning.

[7]  Katsuhiko Ishiguro,et al.  Graph Residual Flow for Molecular Graph Generation , 2019, ArXiv.

[8]  Irena Holubová,et al.  Graph Generators: State of the Art and Open Challenges. , 2020 .

[9]  Jianfeng Gao,et al.  Deep Reinforcement Learning for Dialogue Generation , 2016, EMNLP.

[10]  Wenwu Zhu,et al.  Deep Learning on Graphs: A Survey , 2018, IEEE Transactions on Knowledge and Data Engineering.

[11]  Jiawei Han,et al.  gSpan: graph-based substructure pattern mining , 2002, 2002 IEEE International Conference on Data Mining, 2002. Proceedings..

[12]  Kurt Mehlhorn,et al.  Weisfeiler-Lehman Graph Kernels , 2011, J. Mach. Learn. Res..

[13]  Haoze Wu,et al.  Learning to Generate Industrial SAT Instances , 2021, SOCS.

[14]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[15]  Razvan Pascanu,et al.  Learning Deep Generative Models of Graphs , 2018, ICLR 2018.

[16]  Yoshua Bengio,et al.  DEFactor: Differentiable Edge Factorization-based Probabilistic Graph Generation , 2018, ArXiv.

[17]  Alan Aspuru-Guzik,et al.  Graph Deconvolutional Generation , 2020, ArXiv.

[18]  Rob Fergus,et al.  Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks , 2015, NIPS.

[19]  Renjie Liao,et al.  Efficient Graph Generation with Graph Recurrent Attention Networks , 2019, NeurIPS.

[20]  Jure Leskovec,et al.  How Powerful are Graph Neural Networks? , 2018, ICLR.

[21]  Jing Liu,et al.  Learn to Generate Time Series Conditioned Graphs with Generative Adversarial Nets , 2020, ArXiv.

[22]  Padraig Corcoran,et al.  Deep Q-Learning for Directed Acyclic Graph Generation , 2019, ArXiv.

[23]  Eric Jonas,et al.  Deep imitation learning for molecular inverse problems , 2019, NeurIPS.

[24]  M. Clara De Paolis Kaluza.  A Neural Framework for Learning DAG to DAG Translation , 2018 .

[25]  Aviral Kumar,et al.  Graph Normalizing Flows , 2019, NeurIPS.

[26]  David Duvenaud,et al.  Invertible Residual Networks , 2018, ICML.

[27]  Samy Bengio,et al.  Generating Sentences from a Continuous Space , 2015, CoNLL.

[28]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[29]  Jimeng Sun,et al.  Fast Random Walk Graph Kernel , 2012, SDM.

[30]  Matthias Rarey,et al.  On the Art of Compiling and Using 'Drug‐Like' Chemical Fragment Spaces , 2008, ChemMedChem.

[31]  Ambuj K. Singh,et al.  GraphSig: A Scalable Approach to Mining Significant Subgraphs in Large Graph Databases , 2009, 2009 IEEE 25th International Conference on Data Engineering.

[32]  David Weininger,et al.  SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules , 1988, J. Chem. Inf. Comput. Sci..

[33]  Nasser M. Nasrabadi,et al.  Pattern Recognition and Machine Learning , 2006, Technometrics.

[34]  P. Bonacich.  Power and Centrality: A Family of Measures , 1987, American Journal of Sociology.

[35]  Xavier Bresson,et al.  A Two-Step Graph Convolutional Decoder for Molecule Generation , 2019, ArXiv.

[36]  Davide Bacciu,et al.  Graph generation by sequential edge prediction , 2019, ESANN.

[37]  Michael Guarino.  DiPol-GAN : Generating Molecular Graphs Adversarially with Relational Differentiable Pooling , 2019 .

[38]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[39]  Wenhan Shi,et al.  Conditional Structure Generation through Graph Variational Generative Adversarial Nets , 2019, NeurIPS.

[40]  Ah Chung Tsoi,et al.  The Graph Neural Network Model , 2009, IEEE Transactions on Neural Networks.

[41]  Yang Song,et al.  Generative Modeling by Estimating Gradients of the Data Distribution , 2019, NeurIPS.

[42]  Yanfang Ye,et al.  Interpretable Deep Graph Generation with Node-edge Co-disentanglement , 2020, KDD.

[43]  Svetha Venkatesh,et al.  Graph Transformation Policy Network for Chemical Reaction Prediction , 2018, KDD.

[44]  Li Li,et al.  Decoding Molecular Graph Embeddings with Reinforcement Learning , 2019, ArXiv.

[45]  Xianpei Han,et al.  Sequence-to-Action: End-to-End Semantic Graph Generation for Semantic Parsing , 2018, ACL.

[46]  Sang-Yeon Hwang,et al.  Scaffold-based molecular design using graph generative model , 2019, ArXiv.

[47]  Xiaojie Guo,et al.  Local Event Forecasting and Synthesis Using Unpaired Deep Graph Translations , 2018, LENS@SIGSPATIAL.

[48]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[49]  Harris Chan,et al.  Auto-regressive Graph Generation Modeling with Improved Evaluation Methods , 2019 .

[50]  Ben Poole,et al.  Categorical Reparametrization with Gumble-Softmax , 2017, ICLR 2017.

[51]  Namrata Anand,et al.  Generative modeling for protein structures , 2018, NeurIPS.

[52]  Timo Aila,et al.  A Style-Based Generator Architecture for Generative Adversarial Networks , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[53]  Houman Homayoun,et al.  Deep Multi-attributed Graph Translation with Node-Edge Co-Evolution , 2019, 2019 IEEE International Conference on Data Mining (ICDM).

[54]  Achraf Oussidi,et al.  Deep generative models: Survey , 2018, 2018 International Conference on Intelligent Systems and Computer Vision (ISCV).

[55]  Matt J. Kusner,et al.  Grammar Variational Autoencoder , 2017, ICML.

[56]  Olexandr Isayev,et al.  MolecularRNN: Generating realistic molecular graphs with optimized properties , 2019, ArXiv.

[57]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[58]  Christian Wachinger,et al.  Likelihood-Free Inference and Generation of Molecular Graphs , 2019, ArXiv.

[59]  Weinan Zhang,et al.  GraphAF: a Flow-based Autoregressive Model for Molecular Graph Generation , 2020, ICLR.

[60]  Samy Bengio,et al.  Density estimation using Real NVP , 2016, ICLR.

[61]  Stephan Günnemann,et al.  NetGAN: Generating Graphs via Random Walks , 2018, ICML.

[62]  Simon Osindero,et al.  Conditional Generative Adversarial Nets , 2014, ArXiv.

[63]  Alan M. Frieze,et al.  Random graphs , 2006, SODA '06.

[64]  L. Freeman.  Centrality in social networks conceptual clarification , 1978 .

[65]  Percy Liang,et al.  Data Recombination for Neural Semantic Parsing , 2016, ACL.

[66]  Minyi Guo,et al.  GraphGAN: Graph Representation Learning with Generative Adversarial Nets , 2017, AAAI.

[67]  Andreas Spitz,et al.  DeepNC: Deep Generative Network Completion , 2019, IEEE transactions on pattern analysis and machine intelligence.

[68]  Quoc V. Le,et al.  Distributed Representations of Sentences and Documents , 2014, ICML.

[69]  Anh Tuan Nguyen,et al.  Graph-Based Statistical Language Model for Code , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[70]  Xiaojie Guo,et al.  Deep Graph Translation , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[71]  David Eppstein,et al.  Studying Geometric Graph Properties of Road Networks Through an Algorithmic Lens , 2008 .

[72]  Michalis Vazirgiannis,et al.  Gravity-Inspired Graph Autoencoders for Directed Link Prediction , 2019, CIKM.

[73]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[74]  Kevin Duh,et al.  AMR Parsing as Sequence-to-Graph Transduction , 2019, ACL.

[75]  Seonghyeon Nam,et al.  Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Language , 2018, NeurIPS.

[76]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[77]  Jure Leskovec,et al.  G2SAT: Learning to Generate SAT Formulas , 2019, NeurIPS.

[78]  Motoki Abe,et al.  GraphNVP: An Invertible Flow Model for Generating Molecular Graphs , 2019, ArXiv.

[79]  Markus Weimer,et al.  Learning To Solve Circuit-SAT: An Unsupervised Differentiable Approach , 2018, ICLR.

[80]  Mirella Lapata,et al.  Semantic graph parsing with recurrent neural network DAG grammars , 2019, EMNLP.

[81]  Roman Garnett,et al.  D-VAE: A Variational Autoencoder for Directed Acyclic Graphs , 2019, NeurIPS.

[82]  Gilad Lerman,et al.  Encoding robust representation for graph generation , 2018, 2019 International Joint Conference on Neural Networks (IJCNN).

[83]  Albert-László Barabási,et al.  Statistical mechanics of complex networks , 2001, ArXiv.

[84]  Davide Bacciu,et al.  A Deep Generative Model for Fragment-Based Molecule Generation , 2020, AISTATS.

[85]  Nicola De Cao,et al.  MolGAN: An implicit generative model for small molecular graphs , 2018, ArXiv.

[86]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[87]  Yibo Li,et al.  Multi-objective de novo drug design with conditional graph generative model , 2018, Journal of Cheminformatics.

[88]  Liming Zhang.  STGGAN: Spatial-temporal Graph Generation , 2019, SIGSPATIAL/GIS.

[89]  Stefano Ermon,et al.  Permutation Invariant Graph Generation via Score-Based Generative Modeling , 2020, AISTATS.

[90]  Jure Leskovec,et al.  Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation , 2018, NeurIPS.

[91]  Samuel S. Schoenholz,et al.  Neural Message Passing for Quantum Chemistry , 2017, ICML.

[92]  Peng Wang,et al.  Recent developments in exponential random graph (p*) models for social networks , 2007, Soc. Networks.

[93]  Regina Barzilay,et al.  Junction Tree Variational Autoencoder for Molecular Graph Generation , 2018, ICML.

[94]  Regina Barzilay,et al.  Composing Molecules with Multiple Property Constraints , 2020, ICML 2020.

[95]  Lauwerens Kuipers,et al.  Uniform distribution of sequences , 1974 .

[96]  Christos Faloutsos,et al.  Kronecker Graphs: An Approach to Modeling Networks , 2008, J. Mach. Learn. Res..

[97]  Sebastian Nowozin,et al.  Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observations , 2017, AAAI.

[98]  Max Welling,et al.  Variational Graph Auto-Encoders , 2016, ArXiv.

[99]  Sebastian Fischmeister,et al.  Impact of Community Structure on SAT Solver Performance , 2014, SAT.

[100]  Bruce W. Suter,et al.  The multilayer perceptron as an approximation to a Bayes optimal discriminant function , 1990, IEEE Trans. Neural Networks.

[101]  Geoffrey E. Hinton,et al.  Generating Text with Recurrent Neural Networks , 2011, ICML.

[102]  Vinicius Caridá,et al.  Can NetGAN be Improved on Short Random Walks? , 2019, 2019 8th Brazilian Conference on Intelligent Systems (BRACIS).

[103]  Palash Goyal,et al.  Graph Embedding Techniques, Applications, and Performance: A Survey , 2017, Knowl. Based Syst..

[104]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[105]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[106]  Jason Dykes,et al.  Exploring spatial data representation with dynamic graphics , 1997 .

[107]  Huaiyu Zhu.  On Information and Sufficiency , 1997 .

[108]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[109]  Alessandro Giuliani,et al.  A generative model for protein contact networks , 2015, Journal of biomolecular structure & dynamics.

[110]  Ivan Titov,et al.  AMR Parsing as Graph Prediction with Latent Alignment , 2018, ACL.

[111]  Xiaojie Guo,et al.  Generating Tertiary Protein Structures via an Interpretative Variational Autoencoder , 2020, ArXiv.

[112]  Richard S. Sutton,et al.  Reinforcement Learning of Local Shape in the Game of Go , 2007, IJCAI.

[113]  Marcin J. Skwark,et al.  Protein contact prediction from amino acid co-evolution using convolutional networks for graph-valued images , 2016, NIPS.

[114]  Greg Mori,et al.  Graph Generation with Variational Recurrent Neural Network , 2019, ArXiv.

[115]  Yee Whye Teh,et al.  The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables , 2016, ICLR.

[116]  Honglak Lee,et al.  Learning Structured Output Representation using Deep Conditional Generative Models , 2015, NIPS.

[117]  Olgica Milenkovic,et al.  Multi-MotifGAN (MMGAN): Motif-Targeted Graph Generation And Prediction , 2019, ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[118]  Joshua B. Tenenbaum,et al.  Building machines that learn and think like people , 2016, Behavioral and Brain Sciences.

[119]  Noah A. Smith,et al.  Transition-Based Dependency Parsing with Stack Long Short-Term Memory , 2015, ACL.

[120]  Cao Xiao,et al.  Constrained Generation of Semantically Valid Graphs via Regularizing Variational Autoencoders , 2018, NeurIPS.

[121]  Lukás Burget,et al.  Recurrent neural network based language model , 2010, INTERSPEECH.

[122]  Yu Zhang,et al.  Simple Recurrent Units for Highly Parallelizable Recurrence , 2017, EMNLP.

[123]  Ping Li,et al.  Graph to Graph: a Topology Aware Approach for Graph Structures Learning and Generation , 2019, AISTATS.

[124]  Ulrik Brandes,et al.  What is network science? , 2013, Network Science.

[125]  Cesare Furlanello,et al.  The HIM glocal metric and kernel for network comparison and classification , 2012, 2015 IEEE International Conference on Data Science and Advanced Analytics (DSAA).

[126]  Niloy Ganguly,et al.  NeVAE: A Deep Generative Model for Molecular Graphs , 2018, AAAI.

[127]  Regina Barzilay,et al.  Multi-Objective Molecule Generation using Interpretable Substructures , 2020, ICML.

[128]  Bernhard Schölkopf,et al.  A Kernel Two-Sample Test , 2012, J. Mach. Learn. Res..

[129]  Steven Skiena,et al.  Syntax-Directed Variational Autoencoder for Structured Data , 2018, ICLR.

[130]  Qi Liu,et al.  Constrained Graph Variational Autoencoders for Molecule Design , 2018, NeurIPS.

[131]  Jure Leskovec,et al.  GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Models , 2018, ICML.

[132]  Zhaoyu Wang,et al.  Deep Generative Graph Distribution Learning for Synthetic Power Grids , 2019 .

[133]  Eduardo F. Morales,et al.  An Introduction to Reinforcement Learning , 2011 .

[134]  Nikos Komodakis,et al.  GraphVAE: Towards Generation of Small Graphs Using Variational Autoencoders , 2018, ICANN.

[135]  Marc Brockschmidt,et al.  Disentangling Interpretable Generative Parameters of Random and Real-World Graphs , 2019, ArXiv.

[136]  Ghassan Hamarneh,et al.  BrainNetCNN: Convolutional neural networks for brain networks; towards predicting neurodevelopment , 2017, NeuroImage.

[137]  Kaushalya Madhawa,et al.  GraphNVP: an Invertible Flow-based Model for Generating Molecular Graphs , 2019 .

[138]  Davide Bacciu,et al.  Edge-based sequential graph generation with recurrent neural networks , 2020, Neurocomputing.

[139]  Matthias Rarey,et al.  Feature trees: A new molecular similarity measure based on tree matching , 1998, J. Comput. Aided Mol. Des..

[140]  Jingrui He,et al.  Misc-GAN: A Multi-scale Generative Model for Graphs , 2019, Front. Big Data.

[141]  Li Li,et al.  Optimization of Molecules via Deep Reinforcement Learning , 2018, Scientific Reports.

[142]  Jesús Giráldez-Cru,et al.  A Modularity-Based Random SAT Instances Generator , 2015, IJCAI.

[143]  Daniel Tarlow,et al.  Structured Generative Models of Natural Source Code , 2014, ICML.

[144]  Philip S. Yu,et al.  A Comprehensive Survey on Graph Neural Networks , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[145]  Stefano Ermon,et al.  Graphite: Iterative Generative Modeling of Graphs , 2018, ICML.

[146]  Bert Huang,et al.  Labeled Graph Generative Adversarial Networks , 2019, ArXiv.

[147]  Oleksandr Polozov,et al.  Generative Code Modeling with Graphs , 2018, ICLR.

[148]  Chenliang Xu,et al.  Deep Cross-Modal Audio-Visual Generation , 2017, ACM Multimedia.

[149]  Emmanuel Noutahi,et al.  Towards Interpretable Sparse Graph Representation Learning with Laplacian Pooling , 2019, ArXiv.

[150]  Mirella Lapata,et al.  Language to Logical Form with Neural Attention , 2016, ACL.