Bringing Your Own View: Graph Contrastive Learning without Prefabricated Data Augmentations

Self-supervision is recently surging at its new frontier of graph learning. It facilitates graph representations beneficial to downstream tasks; but its success could hinge on domain knowledge for handcraft or the often expensive trials and errors. Even its stateof-the-art representative, graph contrastive learning (GraphCL), is not completely free of those needs as GraphCL uses a prefabricated prior reflected by the ad-hoc manual selection of graph data augmentations. Our work aims at advancing GraphCL by answering the following questions: How to represent the space of graph augmented views? What principle can be relied upon to learn a prior in that space? And what framework can be constructed to learn the prior in tandem with contrastive learning? Accordingly, we have extended the prefabricated discrete prior in the augmentation set, to a learnable continuous prior in the parameter space of graph generators, assuming that graph priors per se, similar to the concept of image manifolds, can be learned by data generation. Furthermore, to form contrastive views without collapsing to trivial solutions due to the prior learnability, we have leveraged both principles of information minimization (InfoMin) and information bottleneck (InfoBN) to regularize the learned priors. Eventually, contrastive learning, InfoMin, and InfoBN are incorporated organically into one framework of bi-level optimization. Our principled and automated approach has proven to be competitive against the state-of-the-art graph self-supervision methods, including GraphCL, on benchmarks of small graphs; and shown even better generalizability on large-scale graphs, without resorting to human expertise or downstream validation. Our code is publicly released at https://github.com/Shen-Lab/GraphCL_Automated.

[1]  Vijay S. Pande,et al.  MoleculeNet: a benchmark for molecular machine learning , 2017, Chemical science.

[2]  Cordelia Schmid,et al.  What makes for good views for contrastive learning , 2020, NeurIPS.

[3]  Wei Cheng,et al.  InfoGCL: Information-Aware Graph Contrastive Learning , 2021, NeurIPS.

[4]  Suhang Wang,et al.  Self-supervised Learning on Graphs: Deep Insights and New Direction , 2020, ArXiv.

[5]  Yizhou Sun,et al.  Are Powerful Graph Neural Nets Necessary? A Dissection on Graph Classification , 2019, ArXiv.

[6]  Alice H. Oh,et al.  How to Find Your Friendly Neighborhood: Graph Attention Design with Self-Supervision , 2022, ICLR.

[7]  Weinan Zhang,et al.  GraphAF: a Flow-based Autoregressive Model for Molecular Graph Generation , 2020, ICLR.

[8]  Alessandro Rozza,et al.  Graph-Based Neural Network Models with Multiple Self-Supervised Auxiliary Tasks , 2020, Pattern Recognit. Lett..

[9]  M Ashraful Amin,et al.  Node Embedding using Mutual Information and Self-Supervision based Bi-level Aggregation , 2021, 2021 International Joint Conference on Neural Networks (IJCNN).

[10]  Shuiwang Ji,et al.  Self-Supervised Learning of Graph Neural Networks: A Unified Review , 2021, ArXiv.

[11]  Jure Leskovec,et al.  Strategies for Pre-training Graph Neural Networks , 2020, ICLR.

[12]  S. Varadhan,et al.  Asymptotic evaluation of certain Markov process expectations for large time , 1975 .

[13]  Regina Barzilay,et al.  Junction Tree Variational Autoencoder for Molecular Graph Generation , 2018, ICML.

[14]  Alexander A. Alemi,et al.  Deep Variational Information Bottleneck , 2017, ICLR.

[15]  Yatao Bian,et al.  Self-Supervised Graph Transformer on Large-Scale Molecular Data , 2020, NeurIPS.

[16]  Jung-Woo Ha,et al.  Self-supervised Auxiliary Learning with Meta-paths for Heterogeneous Graphs , 2020, NeurIPS.

[17]  Jure Leskovec,et al.  GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Models , 2018, ICML.

[18]  Pietro Liò,et al.  Deep Graph Infomax , 2018, ICLR.

[19]  Jie Zhou,et al.  Distance-wise Graph Contrastive Learning , 2020, ArXiv.

[20]  E. Xing,et al.  Iterative Graph Self-Distillation , 2020, IEEE Transactions on Knowledge and Data Engineering.

[21]  Liefeng Bo,et al.  Heterogeneous Deep Graph Infomax , 2019, ArXiv.

[22]  Yuning You,et al.  Cross-Modality Protein Embedding for Compound-Protein Affinity and Contact Prediction , 2020, bioRxiv.

[23]  Mykola Pechenizkiy,et al.  Hop-Count Based Self-Supervised Anomaly Detection on Attributed Networks , 2021, ArXiv.

[24]  Minyi Guo,et al.  GraphGAN: Graph Representation Learning with Generative Adversarial Nets , 2017, AAAI.

[25]  Marinka Zitnik,et al.  Representation Learning for Networks in Biology and Medicine: Advancements, Challenges, and Opportunities , 2021, ArXiv.

[26]  Ali Ahmed,et al.  Solving Bilinear Inverse Problems using Deep Generative Priors , 2018, ArXiv.

[27]  Truong Q. Nguyen,et al.  Correction by Projection: Denoising Images with Generative Adversarial Networks , 2018, ArXiv.

[28]  Qiang Liu,et al.  Graph Contrastive Learning with Adaptive Augmentation , 2020, WWW.

[29]  Jennifer Neville,et al.  Adversarial Graph Augmentation to Improve Graph Contrastive Learning , 2021, NeurIPS.

[30]  Naftali Tishby,et al.  The information bottleneck method , 2000, ArXiv.

[31]  Minnan Luo,et al.  Self-Supervised Graph Representation Learning via Global Context Prediction , 2020, ArXiv.

[32]  J. Leskovec,et al.  Open Graph Benchmark: Datasets for Machine Learning on Graphs , 2020, NeurIPS.

[33]  Ziniu Hu,et al.  Motif-Driven Contrastive Learning of Graph Representations , 2020, AAAI.

[34]  Philip S. Yu,et al.  Graph Self-Supervised Learning: A Survey , 2021, ArXiv.

[35]  Yixin Chen,et al.  Link Prediction Based on Graph Neural Networks , 2018, NeurIPS.

[36]  Yu Rong,et al.  Graph Information Bottleneck for Subgraph Recognition , 2020, ICLR.

[37]  Yoshua Bengio,et al.  GraphMix: Regularized Training of Graph Neural Networks for Semi-Supervised Learning , 2019, ArXiv.

[38]  Jure Leskovec,et al.  Graph Information Bottleneck , 2020, NeurIPS.

[39]  Chinmay Hegde,et al.  Solving Linear Inverse Problems Using Gan Priors: An Algorithm with Provable Guarantees , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[40]  Yuxiao Dong,et al.  GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training , 2020, KDD.

[41]  Rama Chellappa,et al.  Task-Aware Compressed Sensing with Generative Adversarial Networks , 2018, AAAI.

[42]  Stefano Ermon,et al.  Amortized Variational Compressive Sensing , 2018 .

[43]  Yizhou Sun,et al.  GPT-GNN: Generative Pre-Training of Graph Neural Networks , 2020, KDD.

[44]  Eric Xing,et al.  Prototypical Graph Contrastive Learning , 2021, ArXiv.

[45]  Kristian Kersting,et al.  TUDataset: A collection of benchmark datasets for learning with graphs , 2020, ArXiv.

[46]  Jie Tang,et al.  CODE: Contrastive Pre-training with Adversarial Fine-Tuning for Zero-Shot Expert Linking , 2020, AAAI.

[47]  Jinyoung Park,et al.  Self-supervised Auxiliary Learning for Graph Neural Networks via Meta-Learning , 2021, ArXiv.

[48]  Rob Fergus,et al.  Stochastic Video Generation with a Learned Prior , 2018, ICML.

[49]  Vincent Fortuin,et al.  Priors in Bayesian Deep Learning: A Review , 2021, ArXiv.

[50]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[51]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[52]  L. Wasserman,et al.  The Selection of Prior Distributions by Formal Rules , 1996 .

[53]  Jiliang Tang,et al.  Automated Self-Supervised Learning for Graphs , 2021, ArXiv.

[54]  Vladislav Voroninski,et al.  Global Guarantees for Enforcing Deep Generative Priors by Empirical Risk , 2017, IEEE Transactions on Information Theory.

[55]  Shuiwang Ji,et al.  GraphDF: A Discrete Flow Model for Molecular Graph Generation , 2021, ICML.

[56]  Jure Leskovec,et al.  How Powerful are Graph Neural Networks? , 2018, ICLR.

[57]  Yizhou Sun,et al.  Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networks , 2019, NeurIPS.

[58]  Hong Chen,et al.  Pre-Training Graph Neural Networks for Cold-Start Users and Items Representation , 2020, WSDM.

[59]  Sundeep Rangan,et al.  Inference in Deep Networks in High Dimensions , 2017, 2018 IEEE International Symposium on Information Theory (ISIT).

[60]  Max Welling,et al.  Variational Graph Auto-Encoders , 2016, ArXiv.

[61]  Christos Faloutsos,et al.  Graph mining: Laws, generators, and algorithms , 2006, CSUR.

[62]  Andrea Vedaldi,et al.  Deep Image Prior , 2017, International Journal of Computer Vision.

[63]  Zhangyang Wang,et al.  Graph Contrastive Learning Automated , 2021, ICML.

[64]  Steven Skiena,et al.  DeepWalk: online learning of social representations , 2014, KDD.

[65]  Tianlong Chen,et al.  When Does Self-Supervision Help Graph Convolutional Networks? , 2020, ICML.

[66]  Meng Liu,et al.  GraphEBM: Molecular Graph Generation with Energy-Based Models , 2021, ArXiv.

[67]  Michele Lenza,et al.  Prior Selection for Vector Autoregressions , 2012, Review of Economics and Statistics.

[68]  Leonardo Neves,et al.  Data Augmentation for Graph Neural Networks , 2021, AAAI.

[69]  Minnan Luo,et al.  Graph Representation Learning via Graphical Mutual Information Maximization , 2020, WWW.

[70]  Zhanxing Zhu,et al.  Multi-Stage Self-Supervised Learning for Graph Convolutional Networks , 2020, AAAI.

[71]  John J. Irwin,et al.  ZINC 15 – Ligand Discovery for Everyone , 2015, J. Chem. Inf. Model..

[72]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[73]  Zhangyang Wang,et al.  Graph Contrastive Learning with Augmentations , 2020, NeurIPS.

[74]  Geoffrey E. Hinton,et al.  A Simple Framework for Contrastive Learning of Visual Representations , 2020, ICML.

[75]  Yoshua Bengio,et al.  Mutual Information Neural Estimation , 2018, ICML.

[76]  Yang Liu,et al.  graph2vec: Learning Distributed Representations of Graphs , 2017, ArXiv.

[77]  Stephan Günnemann,et al.  NetGAN: Generating Graphs via Random Walks , 2018, ICML.

[78]  Kaveh Hassani,et al.  Contrastive Multi-View Representation Learning on Graphs , 2020, ICML.

[79]  Jian Tang,et al.  InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization , 2019, ICLR.

[80]  Jie Tang,et al.  Self-Supervised Learning: Generative or Contrastive , 2020, IEEE Transactions on Knowledge and Data Engineering.

[81]  Mingzhe Wang,et al.  LINE: Large-scale Information Network Embedding , 2015, WWW.

[82]  Bernard Ghanem,et al.  FLAG: Adversarial Data Augmentation for Graph Neural Networks , 2020, ArXiv.

[83]  M. Fardad,et al.  Towards A Unified Min-Max Framework for Adversarial Exploration and Robustness , 2019 .

[84]  Donghyun Kim,et al.  Unsupervised Attributed Multiplex Network Embedding , 2020, AAAI.

[85]  David van Dijk,et al.  Self-supervised edge features for improved Graph Neural Network training , 2020, ArXiv.

[86]  Michal Valko,et al.  Bootstrapped Representation Learning on Graphs , 2021, ArXiv.

[87]  Alexandros G. Dimakis,et al.  Compressed Sensing using Generative Models , 2017, ICML.

[88]  Xiangliang Zhang,et al.  Self-Supervised Multi-Channel Hypergraph Convolutional Network for Social Recommendation , 2021, ArXiv.