Graph Attention Multi-Layer Perceptron

Graph neural networks (GNNs) have recently achieved state-of-the-art performance in many graph-based applications. Despite the high performance, they typically need to perform expensive recursive neighborhood expansions during every training epoch and has the scalability issue. Moreover, most of them are inflexible since they are restricted to fixed-hop neighborhoods and insensitive to actual receptive field requirements for each node. We circumvent these limitations by introducing a scalable and flexible method: Graph Attention Multi-Layer Perceptron (GAMLP). Following the routine of decoupled GNNs, the feature propagation in GAMLP is executed during pre-computation, which helps it maintain high scalability. With three proposed receptive field attention, each node in GAMLP is flexible in leveraging the propagated features over the different sizes of reception field. We conduct extensive evaluations on two large open graph benchmarks (ogbnproducts and ogbn-papers100M), demonstrating that GAMLP not only achieves state-of-the-art performance, but also enjoys high scalability and efficiency.

[1]  Chuan Shi,et al.  Learning to Pre-train Graph Neural Networks , 2021, AAAI.

[2]  Chuan Shi,et al.  Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Framework , 2021, WWW.

[3]  Alessandro Rozza,et al.  Graph-Based Neural Network Models with Multiple Self-Supervised Auxiliary Tasks , 2020, Pattern Recognit. Lett..

[4]  Jie Zhou,et al.  Adaptive Graph Encoder for Attributed Graph Embedding , 2020, KDD.

[5]  Jure Leskovec,et al.  OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs , 2021, NeurIPS Datasets and Benchmarks.

[6]  Colin Raffel,et al.  Realistic Evaluation of Deep Semi-Supervised Learning Algorithms , 2018, NeurIPS.

[7]  Francisco Herrera,et al.  Self-labeled techniques for semi-supervised learning: taxonomy, software and empirical study , 2015, Knowledge and Information Systems.

[8]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[9]  Yizhou Sun,et al.  GPT-GNN: Generative Pre-Training of Graph Neural Networks , 2020, KDD.

[10]  Xinliang Wu,et al.  R-GSN: The Relation-based Graph Similar Network for Heterogeneous Graph , 2021, ArXiv.

[11]  Gal Hyams,et al.  Improved Training for Self Training by Confidence Assessments , 2017 .

[12]  Zhanxing Zhu,et al.  Multi-Stage Self-Supervised Learning for Graph Convolutional Networks , 2020, AAAI.

[13]  Xiao-Ming Wu,et al.  Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning , 2018, AAAI.

[14]  Jan Eric Lenssen,et al.  Fast Graph Representation Learning with PyTorch Geometric , 2019, ArXiv.

[15]  Xiao Wang,et al.  Structural Deep Clustering Network , 2020, WWW.

[16]  Qiang Yang,et al.  An Overview of Multi-task Learning , 2018 .

[17]  G. Karypis,et al.  DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphs , 2020, 2020 IEEE/ACM 10th Workshop on Irregular Applications: Architectures and Algorithms (IA3).

[18]  Bowen Du,et al.  Heterogeneous Graph Representation Learning with Relation Awareness , 2021, ArXiv.

[19]  Qian Huang,et al.  Combining Label Propagation and Simple Models Out-performs Graph Neural Networks , 2020, ICLR.

[20]  Dong-Hyun Lee,et al.  Pseudo-Label : The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks , 2013 .

[21]  Yuan He,et al.  Graph Neural Networks for Social Recommendation , 2019, WWW.

[22]  Yu Sun,et al.  Masked Label Prediction: Unified Massage Passing Model for Semi-Supervised Classification , 2020, IJCAI.

[23]  Guoshi Wu,et al.  Scalable and Adaptive Graph Neural Networks with Self-Label-Enhanced training , 2021, ArXiv.

[24]  Zoubin Ghahramani,et al.  Learning from labeled and unlabeled data with label propagation , 2002 .

[25]  Bernard Ghanem,et al.  DeeperGCN: All You Need to Train Deeper GCNs , 2020, ArXiv.

[26]  Simone Scardapane,et al.  Adaptive Propagation Graph Convolutional Network , 2021, IEEE Transactions on Neural Networks and Learning Systems.

[27]  Yong Yu,et al.  Bag of Tricks for Node Classification with Graph Neural Networks , 2021, 2103.13355.

[28]  Cao Xiao,et al.  FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling , 2018, ICLR.

[29]  Max Welling,et al.  Modeling Relational Data with Graph Convolutional Networks , 2017, ESWC.

[30]  Yizhou Sun,et al.  Heterogeneous Graph Transformer , 2020, WWW.

[31]  Le Song,et al.  Stochastic Training of Graph Convolutional Networks with Variance Reduction , 2017, ICML.

[32]  Stephan Günnemann,et al.  Predict then Propagate: Graph Neural Networks meet Personalized PageRank , 2018, ICLR.

[33]  Kilian Q. Weinberger,et al.  Simplifying Graph Convolutional Networks , 2019, ICML.

[34]  Tianlong Chen,et al.  When Does Self-Supervision Help Graph Convolutional Networks? , 2020, ICML.

[35]  Chang Zhou,et al.  AliGraph: A Comprehensive Graph Neural Network Platform , 2019, Proc. VLDB Endow..

[36]  Yikuan Xia,et al.  Evaluating Deep Graph Neural Networks , 2021, ArXiv.

[37]  Samy Bengio,et al.  Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks , 2019, KDD.

[38]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[39]  Davide Eynard,et al.  SIGN: Scalable Inception Graph Neural Networks , 2020, ArXiv.

[40]  Ken-ichi Kawarabayashi,et al.  Representation Learning on Graphs with Jumping Knowledge Networks , 2018, ICML.

[41]  Junzhou Huang,et al.  Adaptive Sampling Towards Fast Graph Representation Learning , 2018, NeurIPS.

[42]  Rajgopal Kannan,et al.  GraphSAINT: Graph Sampling Based Inductive Learning Method , 2019, ICLR.

[43]  Lei Chen,et al.  Reliable Data Distillation on Graph Convolutional Network , 2020, SIGMOD Conference.

[44]  Piotr Koniusz,et al.  Simple Spectral Graph Convolution , 2021, ICLR.

[45]  Weifeng Lv,et al.  Hybrid Micro/Macro Level Convolution for Heterogeneous Graph Learning , 2020, ArXiv.

[46]  Guillaume Bouchard,et al.  Knowledge Graph Completion via Complex Tensor Factorization , 2017, J. Mach. Learn. Res..

[47]  Yaliang Li,et al.  Simple and Deep Graph Convolutional Networks , 2020, ICML.

[48]  Lingfan Yu,et al.  Scalable Graph Neural Networks for Heterogeneous Graphs , 2020, ArXiv.

[49]  Xupeng Miao,et al.  ROD: Reception-aware Online Distillation for Sparse Graphs , 2021, KDD.