Learning How to Propagate Messages in Graph Neural Networks

This paper studies the problem of learning message propagation strategies for graph neural networks (GNNs). One of the challenges for graph neural networks is that of defining the propagation strategy. For instance, the choices of propagation steps are often specialized to a single graph and are not personalized to different nodes. To compensate for this, in this paper, we present learning to propagate, a general learning framework that not only learns the GNN parameters for prediction but more importantly, can explicitly learn the interpretable and personalized propagate strategies for different nodes and various types of graphs. We introduce the optimal propagation steps as latent variables to help find the maximum-likelihood estimation of the GNN parameters in a variational Expectation- Maximization (VEM) framework. Extensive experiments on various types of graph benchmarks demonstrate that our proposed frame- work can significantly achieve better performance compared with the state-of-the-art methods, and can effectively learn personalized and interpretable propagate strategies of messages in GNNs.

[1]  Prasenjit Mitra,et al.  Investigating and Mitigating Degree-Related Biases in Graph Convoltuional Networks , 2020, CIKM.

[2]  Kilian Q. Weinberger,et al.  Simplifying Graph Convolutional Networks , 2019, ICML.

[3]  Paolo Frasconi,et al.  Bilevel Programming for Hyperparameter Optimization and Meta-Learning , 2018, ICML.

[4]  Taiji Suzuki,et al.  Graph Neural Networks Exponentially Lose Expressive Power for Node Classification , 2019, ICLR.

[5]  Massimiliano Pontil,et al.  Learning Discrete Structures for Graph Neural Networks , 2019, ICML.

[6]  Jiliang Tang,et al.  Node Similarity Preserving Graph Convolutional Networks , 2020, WSDM.

[7]  David M. Blei,et al.  Variational Inference: A Review for Statisticians , 2016, ArXiv.

[8]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[9]  Shakir Mohamed,et al.  Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning , 2015, NIPS.

[10]  Tingyang Xu,et al.  DropEdge: Towards Deep Graph Convolutional Networks on Node Classification , 2020, ICLR.

[11]  Satinder Singh,et al.  On Learning Intrinsic Rewards for Policy Gradient Methods , 2018, NeurIPS.

[12]  Yang Gao,et al.  GraphNAS: Graph Neural Architecture Search with Reinforcement Learning , 2019, ArXiv.

[13]  Steven C. H. Hoi,et al.  Towards Locality-Aware Meta-Learning of Tail Node Embeddings on Networks , 2020, CIKM.

[14]  Teng Xiao,et al.  A General Offline Reinforcement Learning Framework for Interactive Recommendation , 2021, AAAI.

[15]  Zaiqiao Meng,et al.  Bayesian Deep Collaborative Matrix Factorization , 2019, AAAI.

[16]  Donglin Wang,et al.  Pareto Self-Supervised Training for Few-Shot Learning , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Suhang Wang,et al.  Say No to the Discrimination: Learning Fair Graph Neural Networks with Limited Sensitive Attribute Information. , 2020 .

[18]  Xiao-Ming Wu,et al.  Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning , 2018, AAAI.

[19]  Xavier Bresson,et al.  Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering , 2016, NIPS.

[20]  Ben Poole,et al.  Categorical Reparameterization with Gumbel-Softmax , 2016, ICLR.

[21]  Cao Xiao,et al.  FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling , 2018, ICLR.

[22]  Mohammad Emtiyaz Khan,et al.  A Stick-Breaking Likelihood for Categorical Data Analysis with Latent Gaussian Models , 2012, AISTATS.

[23]  Philip S. Yu,et al.  A Comprehensive Survey on Graph Neural Networks , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[24]  Xiang Zhang,et al.  GraphSMOTE: Imbalanced Node Classification on Graphs with Graph Neural Networks , 2021, WSDM.

[25]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[26]  Ziqing Xu,et al.  Deep Transfer Tensor Decomposition with Orthogonal Constraint for Recommender Systems , 2021, AAAI.

[27]  Ryan P. Adams,et al.  Gradient-based Hyperparameter Optimization through Reversible Learning , 2015, ICML.

[28]  Bernard Ghanem,et al.  DeepGCNs: Can GCNs Go As Deep As CNNs? , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[29]  Mark W. Schmidt,et al.  Online Learning Rate Adaptation with Hypergradient Descent , 2017, ICLR.

[30]  Sergey Levine,et al.  Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks , 2017, ICML.

[31]  Jung-Woo Ha,et al.  Self-supervised Auxiliary Learning with Meta-paths for Heterogeneous Graphs , 2020, NeurIPS.

[32]  Ruslan Salakhutdinov,et al.  Revisiting Semi-Supervised Learning with Graph Embeddings , 2016, ICML.

[33]  Geoffrey E. Hinton,et al.  The "wake-sleep" algorithm for unsupervised neural networks. , 1995, Science.

[34]  Ken-ichi Kawarabayashi,et al.  Representation Learning on Graphs with Jumping Knowledge Networks , 2018, ICML.

[35]  Leman Akoglu,et al.  PairNorm: Tackling Oversmoothing in GNNs , 2020, ICLR.

[36]  Yaliang Li,et al.  Simple and Deep Graph Convolutional Networks , 2020, ICML.

[37]  L. Akoglu,et al.  Beyond Homophily in Graph Neural Networks: Current Limitations and Effective Designs , 2020, NeurIPS.

[38]  Barak A. Pearlmutter,et al.  Automatic differentiation in machine learning: a survey , 2015, J. Mach. Learn. Res..

[39]  Zaiqiao Meng,et al.  Hierarchical Neural Variational Model for Personalized Sequential Recommendation , 2019, WWW.

[40]  M E J Newman.  Assortative mixing in networks. , 2002, Physical review letters.

[41]  Xiao Huang,et al.  Towards Deeper Graph Neural Networks with Differentiable Group Normalization , 2020, NeurIPS.

[42]  Donglin Wang,et al.  Multi-Initialization Meta-Learning with Domain Adaptation , 2021, ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[43]  Xiao Huang,et al.  Auto-GNN: Neural architecture search of graph neural networks , 2019, Frontiers in Big Data.

[44]  Jie Zhou,et al.  Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological View , 2020, AAAI.

[45]  Yee Whye Teh,et al.  The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables , 2016, ICLR.

[46]  Yann LeCun,et al.  Spectral Networks and Deep Locally Connected Networks on Graphs , 2014 .

[47]  Stephan Günnemann,et al.  Adversarial Attacks on Neural Networks for Graph Data , 2018, KDD.

[48]  Kevin Chen-Chuan Chang,et al.  Geom-GCN: Geometric Graph Convolutional Networks , 2020, ICLR.

[49]  Stephan Günnemann,et al.  Predict then Propagate: Graph Neural Networks meet Personalized PageRank , 2018, ICLR.