Graph Self-Supervised Learning: A Survey

—Deep learning on graphs has attracted significant interests recently. However, most of the works have focused on (semi-) supervised learning, resulting in shortcomings including heavy label reliance, poor generalization, and weak robustness. To address these issues, self-supervised learning (SSL), which extracts informative knowledge through well-designed pretext tasks without relying on manual labels, has become a promising and trending learning paradigm for graph data. Different from SSL on other domains like computer vision and natural language processing, SSL on graphs has an exclusive background, design ideas, and taxonomies. Under the umbrella of graph self-supervised learning , we present a timely and comprehensive review of the existing approaches which employ SSL techniques for graph data. We construct a unified framework that mathematically formalizes the paradigm of graph SSL. According to the objectives of pretext tasks, we divide these approaches into four categories: generation-based, auxiliary property-based, contrast-based, and hybrid approaches. We further describe the applications of graph SSL across various research fields and summarize the commonly used datasets, evaluation benchmark, performance comparison and open-source codes of graph SSL. Finally, we discuss the remaining challenges and potential future directions in this research field.

[1]  Percy Liang,et al.  Graph-based, Self-Supervised Program Repair from Diagnostic Feedback , 2020, ICML.

[2]  Pinar Yanardag,et al.  Deep Graph Kernels , 2015, KDD.

[3]  Fillia Makedon,et al.  A Survey on Contrastive Self-supervised Learning , 2020, Technologies.

[4]  Yizhou Sun,et al.  GPT-GNN: Generative Pre-Training of Graph Neural Networks , 2020, KDD.

[5]  Shirui Pan,et al.  ANEMONE: Graph Anomaly Detection with Multi-Scale Contrastive Learning , 2021, CIKM.

[6]  Mingzhe Wang,et al.  LINE: Large-scale Information Network Embedding , 2015, WWW.

[7]  Shu Wu,et al.  CAGNN: Cluster-Aware Graph Neural Networks for Unsupervised Graph Representation Learning , 2020, ArXiv.

[8]  Lizong Zhang,et al.  Multi-View Attributed Graph Clustering , 2021, IEEE Transactions on Knowledge and Data Engineering.

[9]  Guido Montúfar,et al.  Haar Graph Pooling , 2019, ICML.

[10]  George Karypis,et al.  Comparison of descriptor spaces for chemical compound retrieval and classification , 2006, Sixth International Conference on Data Mining (ICDM'06).

[11]  G. Karypis,et al.  Anomaly Detection on Attributed Networks via Contrastive Self-Supervised Learning , 2021, IEEE Transactions on Neural Networks and Learning Systems.

[12]  Yizhou Sun,et al.  Leveraging Meta-path Contexts for Classification in Heterogeneous Information Networks , 2020, 2021 IEEE 37th International Conference on Data Engineering (ICDE).

[13]  George Karypis,et al.  Graph InfoClust: Maximizing Coarse-Grain Mutual Information in Graphs , 2021, PAKDD.

[14]  Qiang Liu,et al.  Graph Contrastive Learning with Adaptive Augmentation , 2020, WWW.

[15]  Pietro Liò,et al.  Spatio-Temporal Deep Graph Infomax , 2019, ArXiv.

[16]  Xinlei Chen,et al.  Exploring Simple Siamese Representation Learning , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Zhangyang Wang,et al.  Graph Contrastive Learning Automated , 2021, ICML.

[18]  Jianren Wang,et al.  MolCLR: Molecular Contrastive Learning of Representations via Graph Neural Networks , 2021, ArXiv.

[19]  C. Shi,et al.  Adversarial Label-Flipping Attack and Defense for Graph Neural Networks , 2020, 2020 IEEE International Conference on Data Mining (ICDM).

[20]  Elise van der Pol,et al.  Contrastive Learning of Structured World Models , 2019, ICLR.

[21]  Sarunas Girdzijauskas,et al.  Self-supervised Graph Neural Networks without explicit negative sampling , 2021, ArXiv.

[22]  Liefeng Bo,et al.  Heterogeneous Deep Graph Infomax , 2019, ArXiv.

[23]  Matthijs Douze,et al.  Deep Clustering for Unsupervised Learning of Visual Features , 2018, ECCV.

[24]  Peter Willett,et al.  Effectiveness of 2D fingerprints for scaffold hopping. , 2011, Future medicinal chemistry.

[25]  Jie Tang,et al.  CODE: Contrastive Pre-training with Adversarial Fine-Tuning for Zero-Shot Expert Linking , 2020, AAAI.

[26]  Kaveh Hassani,et al.  Contrastive Multi-View Representation Learning on Graphs , 2020, ICML.

[27]  Stan Z. Li,et al.  Self-supervised on Graphs: Contrastive, Generative, or Predictive , 2021, ArXiv.

[28]  Ziniu Hu,et al.  Motif-Driven Contrastive Learning of Graph Representations , 2020, AAAI.

[29]  N. Chawla,et al.  Graph Barlow Twins: A self-supervised representation learning framework for graphs , 2021, Knowl. Based Syst..

[30]  M. McPherson,et al.  Birds of a Feather: Homophily in Social Networks , 2001 .

[31]  Pengtao Xie,et al.  Contrastive Self-supervised Learning for Graph Classification , 2020, AAAI.

[32]  Mykola Pechenizkiy,et al.  Hop-Count Based Self-Supervised Anomaly Detection on Attributed Networks , 2021, ArXiv.

[33]  Zhang Xiong,et al.  Contrastive Learning for Recommender System , 2021, ArXiv.

[34]  Stephan Günnemann,et al.  Diffusion Improves Graph Learning , 2019, NeurIPS.

[35]  Zhangyang Wang,et al.  Graph Contrastive Learning with Augmentations , 2020, NeurIPS.

[36]  Paolo Favaro,et al.  Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles , 2016, ECCV.

[37]  Jian Tang,et al.  InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization , 2019, ICLR.

[38]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[39]  Pietro Liò,et al.  Graph Attention Networks , 2017, ICLR.

[40]  Geoffrey E. Hinton,et al.  A Simple Framework for Contrastive Learning of Visual Representations , 2020, ICML.

[41]  Guangchun Luo,et al.  Self-supervised Consensus Representation Learning for Attributed Graph , 2021, ACM Multimedia.

[42]  Roger Wattenhofer,et al.  Towards Robust Graph Contrastive Learning , 2021, ArXiv.

[43]  Antony J. Williams,et al.  ToxCast Chemical Landscape: Paving the Road to 21st Century Toxicology. , 2016, Chemical research in toxicology.

[44]  Jure Leskovec,et al.  node2vec: Scalable Feature Learning for Networks , 2016, KDD.

[45]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[46]  Jie Tang,et al.  Self-Supervised Learning: Generative or Contrastive , 2020, IEEE Transactions on Knowledge and Data Engineering.

[47]  Chunyan Miao,et al.  Pre-training Graph Transformer with Multimodal Side Information for Recommendation , 2021, ACM Multimedia.

[48]  Pierre H. Richemond,et al.  Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning , 2020, NeurIPS.

[49]  Ruslan Salakhutdinov,et al.  Revisiting Semi-Supervised Learning with Graph Embeddings , 2016, ICML.

[50]  Michael Tschannen,et al.  On Mutual Information Maximization for Representation Learning , 2019, ICLR.

[51]  Vipin Kumar,et al.  Multilevel Graph Partitioning Schemes , 1995, ICPP.

[52]  Zibin Zheng,et al.  FedGL: Federated Graph Learning Framework with Global Self-Supervision , 2021, Inf. Sci..

[53]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[54]  Xin Chen,et al.  Multi-view Graph Contrastive Representation Learning for Drug-Drug Interaction Prediction , 2020, WWW.

[55]  Jundong Li,et al.  SpecAE: Spectral AutoEncoder for Anomaly Detection in Attributed Networks , 2019, CIKM.

[56]  Liang Wang,et al.  Deep Graph Contrastive Representation Learning , 2020, ArXiv.

[57]  Yingli Tian,et al.  Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[58]  Sunyoung Kwon,et al.  Self-supervised Auxiliary Learning with Meta-paths for Heterogeneous Graphs , 2020, NeurIPS.

[59]  Seyed Mehran Kazemi,et al.  SLAPS: Self-Supervision Improves Structure Learning for Graph Neural Networks , 2021, NeurIPS.

[60]  Vijay S. Pande,et al.  SWEETLEAD: an In Silico Database of Approved Drugs, Regulated Chemicals, and Herbal Isolates for Computer-Aided Drug Discovery , 2013, PloS one.

[61]  Suhang Wang,et al.  Self-supervised Learning on Graphs: Deep Insights and New Direction , 2020, ArXiv.

[62]  Ashwin Srinivasan,et al.  Statistical Evaluation of the Predictive Toxicology Challenge 2000-2001 , 2003, Bioinform..

[63]  Jiliang Tang,et al.  Node Similarity Preserving Graph Convolutional Networks , 2020, WSDM.

[64]  Yunyun Wang,et al.  Pre-Training on Dynamic Graph Neural Networks , 2021, Neurocomputing.

[65]  Max Welling,et al.  Variational Graph Auto-Encoders , 2016, ArXiv.

[66]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[67]  Michal Valko,et al.  Bootstrapped Representation Learning on Graphs , 2021, ArXiv.

[68]  J. Leskovec,et al.  Open Graph Benchmark: Datasets for Machine Learning on Graphs , 2020, NeurIPS.

[69]  Carl Yang,et al.  Transfer Learning of Graph Neural Networks with Ego-graph Information Maximization , 2020, NeurIPS.

[70]  O. Iordache,et al.  Technologies , 2018, Studies in Systems, Decision and Control.

[71]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[72]  Hyung Jin Chang,et al.  Symmetric Graph Convolutional Autoencoder for Unsupervised Graph Representation Learning , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[73]  Jure Leskovec,et al.  How Powerful are Graph Neural Networks? , 2018, ICLR.

[74]  Mounir Ghogho,et al.  GraphCL: Contrastive Self-Supervised Learning of Graph Representations , 2020, ArXiv.

[75]  Nguyen Quoc Viet Hung,et al.  Self-Supervised Multi-Channel Hypergraph Convolutional Network for Social Recommendation , 2021, WWW.

[76]  Hanghang Tong,et al.  HDMI: High-order Deep Multiplex Infomax , 2021, WWW.

[77]  Shirui Pan,et al.  Contrastive Graph Poisson Networks: Semi-Supervised Learning with Extremely Limited Labels , 2021, NeurIPS.

[78]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[79]  Ming Jin,et al.  Towards Graph Self-Supervised Learning with Contrastive Adjusted Zooming , 2021, ArXiv.

[80]  S. Sra,et al.  Contrastive Learning with Hard Negative Samples , 2020, ICLR.

[81]  Yun Xiong,et al.  Sub-graph Contrast for Scalable Self-Supervised Graph Representation Learning , 2020, 2020 IEEE International Conference on Data Mining (ICDM).

[82]  Li Sun,et al.  Context Matters: Graph-based Self-supervised Representation Learning for Medical Images , 2020, AAAI.

[83]  Quoc V. Le,et al.  Towards Domain-Agnostic Contrastive Learning , 2020, ICML.

[84]  Jiawei Zhang,et al.  Graph-Bert: Only Attention is Needed for Learning Graph Representations , 2020, ArXiv.

[85]  Alexei A. Efros,et al.  Context Encoders: Feature Learning by Inpainting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[86]  Hong Chen,et al.  Pre-Training Graph Neural Networks for Cold-Start Users and Items Representation , 2020, WSDM.

[87]  Philip S. Yu,et al.  A Comprehensive Survey on Graph Neural Networks , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[88]  Alexei A. Efros,et al.  Colorful Image Colorization , 2016, ECCV.

[89]  Philip S. Yu,et al.  A Survey on Knowledge Graphs: Representation, Acquisition, and Applications , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[90]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[91]  Huan Liu,et al.  Inductive Anomaly Detection on Attributed Networks , 2020, IJCAI.

[92]  Minnan Luo,et al.  Graph Representation Learning via Graphical Mutual Information Maximization , 2020, WWW.

[93]  Yann LeCun,et al.  Barlow Twins: Self-Supervised Learning via Redundancy Reduction , 2021, ICML.

[94]  Alessandro Rozza,et al.  Graph-Based Neural Network Models with Multiple Self-Supervised Auxiliary Tasks , 2020, Pattern Recognit. Lett..

[95]  M Ashraful Amin,et al.  Node Embedding using Mutual Information and Self-Supervision based Bi-level Aggregation , 2021, 2021 International Joint Conference on Neural Networks (IJCNN).

[96]  Shuiwang Ji,et al.  Self-Supervised Learning of Graph Neural Networks: A Unified Review , 2021, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[97]  Shenglan Liu,et al.  Self-Supervised Deep Graph Embedding with High-Order Information Fusion for Community Discovery , 2021, ArXiv.

[98]  P'eter Mernyei,et al.  Wiki-CS: A Wikipedia-Based Benchmark for Graph Neural Networks , 2020, ArXiv.

[99]  Alexander Davis,et al.  Feature Selection for Deep Neural Networks in Cyber Security Applications , 2020, 2020 IEEE International IOT, Electronics and Mechatronics Conference (IEMTRONICS).

[100]  Yuxiao Dong,et al.  GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training , 2020, KDD.

[101]  Jie Zhou,et al.  Distance-wise Graph Contrastive Learning , 2020, ArXiv.

[102]  Heng Tao Shen,et al.  Principal Component Analysis , 2009, Encyclopedia of Biometrics.

[103]  Pietro Liò,et al.  Deep Graph Infomax , 2018, ICLR.

[104]  Ping Wang,et al.  Self-Supervised Learning of Contextual Embeddings for Link Prediction in Heterogeneous Networks , 2020, WWW.

[105]  Phillip Isola,et al.  Contrastive Multiview Coding , 2019, ECCV.

[106]  Guo-Jun Qi,et al.  Contrastive Learning With Stronger Augmentations , 2021, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[107]  Yue Wu,et al.  Maximizing Mutual Information Across Feature and Topology Views for Learning Graph Representations , 2021, ArXiv.

[108]  Wenbo Xu,et al.  Structured Graph Learning for Scalable Subspace Clustering: From Single View to Multiview , 2021, IEEE Transactions on Cybernetics.

[109]  Pengfei Chen,et al.  Improving Graph Representation Learning by Contrastive Regularization , 2021, ArXiv.

[110]  Yatao Bian,et al.  Self-Supervised Graph Transformer on Large-Scale Molecular Data , 2020, NeurIPS.

[111]  Jennifer Neville,et al.  Adversarial Graph Augmentation to Improve Graph Contrastive Learning , 2021, NeurIPS.

[112]  Tianlong Chen,et al.  When Does Self-Supervision Help Graph Convolutional Networks? , 2020, ICML.

[113]  Ross B. Girshick,et al.  Momentum Contrast for Unsupervised Visual Representation Learning , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[114]  Tianyi Zhou,et al.  FedProto: Federated Prototype Learning over Heterogeneous Devices , 2021, ArXiv.

[115]  Chen Gong,et al.  Contrastive and Generative Graph Convolutional Networks for Graph-based Semi-Supervised Learning , 2020, AAAI.

[116]  Oriol Vinyals,et al.  Representation Learning with Contrastive Predictive Coding , 2018, ArXiv.

[117]  J. Leskovec,et al.  Strategies for Pre-training Graph Neural Networks , 2019, ICLR.

[118]  Liefeng Bo,et al.  HDGI: An Unsupervised Graph Neural Network for Representation Learning in Heterogeneous Graph , 2019 .

[119]  Jure Leskovec,et al.  Predicting multicellular function through multi-layer tissue networks , 2017, Bioinform..

[120]  Qi Liu,et al.  Constrained Graph Variational Autoencoders for Molecule Design , 2018, NeurIPS.

[121]  Minnan Luo,et al.  Self-Supervised Graph Representation Learning via Global Context Prediction , 2020, ArXiv.

[122]  Bo Du,et al.  Self-supervised Training of Graph Convolutional Networks , 2020, ArXiv.

[123]  Hannu Toivonen,et al.  Statistical evaluation of the predictive toxicology challenge , 2000 .

[124]  Hans-Peter Kriegel,et al.  Protein function prediction via graph kernels , 2005, ISMB.

[125]  Chenguang Wang,et al.  Learning graph representation by aggregating subgraphs via mutual information maximization , 2021, Neurocomputing.

[126]  Huan Liu,et al.  Deep Anomaly Detection on Attributed Networks , 2019, SDM.

[127]  Philip S. Yu,et al.  A Survey of Community Detection Approaches: From Statistical Modeling to Deep Learning , 2021, IEEE Transactions on Knowledge and Data Engineering.

[128]  Peer Bork,et al.  The SIDER database of drugs and side effects , 2015, Nucleic Acids Res..

[129]  P. Dobson,et al.  Distinguishing enzyme structures from non-enzymes without alignments. , 2003, Journal of molecular biology.

[130]  Stephan Günnemann,et al.  Pitfalls of Graph Neural Network Evaluation , 2018, ArXiv.

[131]  Xiaoning Qian,et al.  Semi-Implicit Graph Variational Auto-Encoders , 2019, NeurIPS.

[132]  Lise Getoor,et al.  Collective Classification in Network Data , 2008, AI Mag..

[133]  Hao Peng,et al.  SUGAR: Subgraph Neural Network with Reinforcement Pooling and Self-Supervised Mutual Information Mechanism , 2021, WWW.

[134]  Vijay S. Pande,et al.  Computational Modeling of β-Secretase 1 (BACE-1) Inhibitors Using Ligand Based Approaches , 2016, J. Chem. Inf. Model..

[135]  Zhao Li,et al.  Hierarchical Bipartite Graph Neural Networks: Towards Large-Scale E-commerce Applications , 2020, 2020 IEEE 36th International Conference on Data Engineering (ICDE).

[136]  E. Xing,et al.  Iterative Graph Self-Distillation , 2020, IEEE Transactions on Knowledge and Data Engineering.

[137]  Bingbing Ni,et al.  Self-supervised Graph-level Representation Learning with Local and Global Structure , 2021, ICML.

[138]  Ashwin Srinivasan,et al.  The Predictive Toxicology Challenge 2000-2001 , 2001, Bioinform..

[139]  Lina Yao,et al.  Adversarially Regularized Graph Autoencoder , 2018, IJCAI.

[140]  Jure Leskovec,et al.  Hierarchical Graph Representation Learning with Differentiable Pooling , 2018, NeurIPS.

[141]  Blaise Agüera y Arcas,et al.  Communication-Efficient Learning of Deep Networks from Decentralized Data , 2016, AISTATS.

[142]  Khoa T. Phan,et al.  Generative and Contrastive Self-Supervised Learning for Graph Anomaly Detection , 2021, IEEE Transactions on Knowledge and Data Engineering.

[143]  Alice H. Oh,et al.  How to Find Your Friendly Neighborhood: Graph Attention Design with Self-Supervision , 2022, ICLR.

[144]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[145]  Vipin Kumar,et al.  A Fast and High Quality Multilevel Scheme for Partitioning Irregular Graphs , 1998, SIAM J. Sci. Comput..

[146]  Yiming Yang,et al.  XLNet: Generalized Autoregressive Pretraining for Language Understanding , 2019, NeurIPS.

[147]  Pascal Vincent,et al.  Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion , 2010, J. Mach. Learn. Res..

[148]  Yuxiang Ren,et al.  Label Contrastive Coding based Graph Neural Network for Graph Classification , 2021, DASFAA.

[149]  Xinjiang Lu,et al.  GraphMS: Drug Target Prediction Using Graph Representation Learning with Substructures , 2021, Applied Sciences.

[150]  Zhouchen Lin,et al.  Multi-Stage Self-Supervised Learning for Graph Convolutional Networks , 2019, AAAI.

[151]  Luis Pinheiro,et al.  A Bayesian Approach to in Silico Blood-Brain Barrier Penetration Modeling , 2012, J. Chem. Inf. Model..

[152]  Chun Wang,et al.  MGAE: Marginalized Graph Autoencoder for Graph Clustering , 2017, CIKM.

[153]  Chuan Shi,et al.  Self-supervised Heterogeneous Graph Neural Network with Co-contrastive Learning , 2021, KDD.

[154]  Junzhou Huang,et al.  DropEdge: Towards Deep Graph Convolutional Networks on Node Classification , 2019, International Conference on Learning Representations.

[155]  Xixun Lin,et al.  Bipartite Graph Embedding via Mutual Information Maximization , 2020, WSDM.

[156]  Yoshua Bengio,et al.  Learning deep representations by mutual information estimation and maximization , 2018, ICLR.

[157]  Jiawei Han,et al.  Unsupervised Attributed Multiplex Network Embedding , 2019, AAAI.

[158]  Yizhou Sun,et al.  Pre-Training Graph Neural Networks for Generic Structural Feature Extraction , 2019, ArXiv.

[159]  George Karypis,et al.  Graph InfoClust: Leveraging cluster-level node information for unsupervised graph representation learning , 2020, ArXiv.

[160]  A. Debnath,et al.  Structure-activity relationship of mutagenic aromatic and heteroaromatic nitro compounds. Correlation with molecular orbital energies and hydrophobicity. , 1991, Journal of medicinal chemistry.

[161]  Huajun Chen,et al.  Knowledge-aware Contrastive Molecular Graph Learning , 2021, ArXiv.

[162]  David van Dijk,et al.  Self-supervised edge features for improved Graph Neural Network training , 2020, ArXiv.

[163]  Xinbo Gao,et al.  Self-supervised graph convolutional clustering by preserving latent distribution , 2021, Neurocomputing.

[164]  Jing Jiang,et al.  Graph WaveNet for Deep Spatial-Temporal Graph Modeling , 2019, IJCAI.

[165]  Charu C. Aggarwal,et al.  Graph Clustering , 2010, Encyclopedia of Machine Learning and Data Mining.

[166]  Steven Skiena,et al.  DeepWalk: online learning of social representations , 2014, KDD.