Improving the Training of Graph Neural Networks with Consistency Regularization

Graph neural networks (GNNs) have achieved notable success in the semi-supervised learning scenario. The message passing mechanism in graph neural networks helps unlabeled nodes gather supervision signals from their labeled neighbors. In this work, we investigate how consistency regularization, one of widely adopted semi-supervised learning methods, can help improve the performance of graph neural networks. We revisit two methods of consistency regularization for graph neural networks. One is simple consistency regularization (SCR), and the other is mean-teacher consistency regularization (MCR). We combine the consistency regularization methods with two state-of-the-art GNNs and conduct experiments on the ogbn-products dataset1. With the consistency regularization, the performance of state-of-the-art GNNs can be improved by 0.3% on the ogbn-products dataset of Open Graph Benchmark (OGB) both with and without external data.

[1]  Cho-Jui Hsieh,et al.  Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction , 2021, ArXiv.

[2]  James Cheng,et al.  Self-Enhanced GNN: Improving Graph Neural Networks Using Model Outputs , 2020, 2021 International Joint Conference on Neural Networks (IJCNN).

[3]  Kilian Q. Weinberger,et al.  Simplifying Graph Convolutional Networks , 2019, ICML.

[4]  Junzhou Huang,et al.  Adaptive Sampling Towards Fast Graph Representation Learning , 2018, NeurIPS.

[5]  David Berthelot,et al.  MixMatch: A Holistic Approach to Semi-Supervised Learning , 2019, NeurIPS.

[6]  Joan Bruna,et al.  Spectral Networks and Locally Connected Networks on Graphs , 2013, ICLR.

[7]  Timo Aila,et al.  Temporal Ensembling for Semi-Supervised Learning , 2016, ICLR.

[8]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[9]  Davide Eynard,et al.  SIGN: Scalable Inception Graph Neural Networks , 2020, ArXiv.

[10]  Xiao-Ming Wu,et al.  Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning , 2018, AAAI.

[11]  Harri Valpola,et al.  Weight-averaged consistency targets improve semi-supervised deep learning results , 2017, ArXiv.

[12]  Cao Xiao,et al.  FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling , 2018, ICLR.

[13]  J. Leskovec,et al.  Open Graph Benchmark: Datasets for Machine Learning on Graphs , 2020, NeurIPS.

[14]  Tingyang Xu,et al.  DropEdge: Towards Deep Graph Convolutional Networks on Node Classification , 2020, ICLR.

[15]  Guoshi Wu,et al.  Scalable and Adaptive Graph Neural Networks with Self-Label-Enhanced training , 2021, ArXiv.

[16]  Yoshua Bengio,et al.  GMNN: Graph Markov Neural Networks , 2019, ICML.

[17]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[18]  Qian Huang,et al.  Combining Label Propagation and Simple Models Out-performs Graph Neural Networks , 2020, ICLR.

[19]  Ziqi Yin,et al.  Graph Attention Multi-Layer Perceptron , 2021, ArXiv.

[20]  Shuiwang Ji,et al.  Towards Deeper Graph Neural Networks , 2020, KDD.

[21]  Rajgopal Kannan,et al.  GraphSAINT: Graph Sampling Based Inductive Learning Method , 2019, ICLR.

[22]  Yongdong Zhang,et al.  LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation , 2020, SIGIR.

[23]  Jure Leskovec,et al.  How Powerful are Graph Neural Networks? , 2018, ICLR.

[24]  Juho Kannala,et al.  GraphMix: Improved Training of GNNs for Semi-Supervised Learning , 2020, AAAI.

[25]  Yang Gao,et al.  GraphNAS: Graph Neural Architecture Search with Reinforcement Learning , 2019, ArXiv.

[26]  Tapani Raiko,et al.  Semi-supervised Learning with Ladder Networks , 2015, NIPS.

[27]  Xiaojin Zhu,et al.  --1 CONTENTS , 2006 .

[28]  O. Chapelle,et al.  Semi-Supervised Learning (Chapelle, O. et al., Eds.; 2006) [Book reviews] , 2009, IEEE Transactions on Neural Networks.

[29]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[30]  Chang Zhou,et al.  CogDL: An Extensive Toolkit for Deep Learning on Graphs , 2021, ArXiv.

[31]  David Berthelot,et al.  ReMixMatch: Semi-Supervised Learning with Distribution Matching and Augmentation Anchoring , 2020, ICLR.

[32]  Qian Xu,et al.  Graph Random Neural Networks for Semi-Supervised Learning on Graphs , 2020, NeurIPS.

[33]  Quoc V. Le,et al.  Unsupervised Data Augmentation for Consistency Training , 2019, NeurIPS.

[34]  Xiaojin Zhu,et al.  Introduction to Semi-Supervised Learning , 2009, Synthesis Lectures on Artificial Intelligence and Machine Learning.