Graph Representation Learning via Contrasting Cluster Assignments

With the rise of contrastive learning, unsupervised graph representation learning has been booming recently, and shown strong competitiveness, even surpassing the supervised counterparts in some machine learning tasks. Most of existing contrastive models for graph representation learning either focus on maximizing mutual information between local and global embeddings, or primarily depend on contrasting embeddings at node level. However, they are still not exquisite enough to comprehensively explore the local and global views of network topology. Although the former considers local-global relationship, its coarse global information leads to grudging cooperation between local and global views. The latter pays attention to node-level feature alignment, so that the role of global view appears inconspicuous. To avoid falling into these two extreme cases, we propose a novel unsupervised graph representation model by contrasting cluster assignments, called as GRCCA, that can keep a balanced aggregation of local and global information. It is motivated to leverage clustering algorithms to grasp the more fine-grained global information (cluster-level), and get insight into the elusive association between nodes beyond graph topology. Moreover, we contrast embeddings at node level to preserve quality of local information, but enforce the cluster-level consistency instead of node-level consistency to explore global information elegantly. Specifically, we first generate two augmented graphs with distinct graph augmentation strategies, then employ clustering algorithms to obtain their cluster assignments and prototypes respectively. The proposed GRCCA further compels the identical nodes from different augmented graphs to recognize their cluster assignments mutually by minimizing a cross entropy loss. To demonstrate its effectiveness, we compare with the state-of-the-art models in three different downstream tasks, including node classification, link prediction and community detection. The experimental results show that GRCCA has strong competitiveness in most tasks.

[1]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[2]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[3]  Minnan Luo,et al.  Graph Representation Learning via Graphical Mutual Information Maximization , 2020, WWW.

[4]  Tianlong Chen,et al.  When Does Self-Supervision Help Graph Convolutional Networks? , 2020, ICML.

[5]  Geoffrey E. Hinton,et al.  A Simple Framework for Contrastive Learning of Visual Representations , 2020, ICML.

[6]  Timothy M. Hospedales,et al.  How Well Do Self-Supervised Models Transfer? , 2020, ArXiv.

[7]  Philip S. Yu,et al.  A Comprehensive Survey on Graph Neural Networks , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[8]  Stephan Günnemann,et al.  Diffusion Improves Graph Learning , 2019, NeurIPS.

[9]  Pietro Liò,et al.  Deep Graph Infomax , 2018, ICLR.

[10]  Jure Leskovec,et al.  Representation Learning on Graphs: Methods and Applications , 2017, IEEE Data Eng. Bull..

[11]  Yu Chen,et al.  Iterative Deep Graph Learning for Graph Neural Networks: Better and Robust Node Embeddings , 2019, NeurIPS.

[12]  Max Welling,et al.  Variational Graph Auto-Encoders , 2016, ArXiv.

[13]  Di He,et al.  Do Transformers Really Perform Bad for Graph Representation? , 2021, ArXiv.

[14]  Stephan Günnemann,et al.  Pitfalls of Graph Neural Network Evaluation , 2018, ArXiv.

[15]  Steven Skiena,et al.  DeepWalk: online learning of social representations , 2014, KDD.

[16]  Seong Joon Oh,et al.  CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[17]  Julien Mairal,et al.  Unsupervised Learning of Visual Features by Contrasting Cluster Assignments , 2020, NeurIPS.

[18]  Chen Gong,et al.  Multi-Scale Contrastive Siamese Networks for Self-Supervised Graph Representation Learning , 2021, IJCAI.

[19]  Graham Cormode,et al.  Node Classification in Social Networks , 2011, Social Network Data Analytics.

[20]  Yingli Tian,et al.  Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Suhang Wang,et al.  Self-supervised Learning on Graphs: Deep Insights and New Direction , 2020, ArXiv.

[22]  Zhiyuan Liu,et al.  Representation Learning of Knowledge Graphs with Entity Descriptions , 2016, AAAI.

[23]  Chengxu Zhuang,et al.  Local Aggregation for Unsupervised Learning of Visual Embeddings , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[24]  Jure Leskovec,et al.  node2vec: Scalable Feature Learning for Networks , 2016, KDD.

[25]  Michal Valko,et al.  Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning , 2020, NeurIPS.

[26]  Andrea Vedaldi,et al.  Self-labelling via simultaneous clustering and representation learning , 2020, ICLR.

[27]  Jon M. Kleinberg,et al.  The link-prediction problem for social networks , 2007, J. Assoc. Inf. Sci. Technol..

[28]  Qiang Liu,et al.  Graph Contrastive Learning with Adaptive Augmentation , 2020, WWW.

[29]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[30]  Xavier Bresson,et al.  A Generalization of Transformer Networks to Graphs , 2020, ArXiv.

[31]  Kaiming He,et al.  Momentum Contrast for Unsupervised Visual Representation Learning , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Jian Pei,et al.  Asymmetric Transitivity Preserving Graph Embedding , 2016, KDD.

[33]  Katarzyna Musial,et al.  Foundations and Modeling of Dynamic Networks Using Dynamic Graph Neural Networks: A Survey , 2020, IEEE Access.

[34]  Pascal Poupart,et al.  Representation Learning for Dynamic Graphs: A Survey , 2020, J. Mach. Learn. Res..

[35]  Xavier Bresson,et al.  Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering , 2016, NIPS.

[36]  Julien Mairal,et al.  Unsupervised Pre-Training of Image Features on Non-Curated Data , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[37]  Matthijs Douze,et al.  Deep Clustering for Unsupervised Learning of Visual Features , 2018, ECCV.

[38]  Jiawei Zhang Graph Neural Networks for Small Graph and Giant Network Representation Learning: An Overview , 2019, ArXiv.

[39]  Pengtao Xie,et al.  Contrastive Self-supervised Learning for Graph Classification , 2020, AAAI.

[40]  Yoshua Bengio,et al.  Learning deep representations by mutual information estimation and maximization , 2018, ICLR.

[41]  Dezhong Peng,et al.  Contrastive Clustering , 2021, AAAI.

[42]  Qiongkai Xu,et al.  GraRep: Learning Graph Representations with Global Structural Information , 2015, CIKM.

[43]  Hasan Şakir Bilge,et al.  Deep Metric Learning: A Survey , 2019, Symmetry.

[44]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[45]  George Karypis,et al.  Graph InfoClust: Leveraging cluster-level node information for unsupervised graph representation learning , 2020, ArXiv.

[46]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[47]  Kaveh Hassani,et al.  Contrastive Multi-View Representation Learning on Graphs , 2020, ICML.

[48]  Qiang Liu,et al.  Deep Graph Contrastive Representation Learning , 2020, ArXiv.

[49]  Hao Peng,et al.  SUGAR: Subgraph Neural Network with Reinforcement Pooling and Self-Supervised Mutual Information Mechanism , 2021, WWW.

[50]  Xiaoning Qian,et al.  Semi-Implicit Graph Variational Auto-Encoders , 2019, NeurIPS.

[51]  Xin Wang,et al.  Deep Learning for Learning Graph Representations , 2019, Deep Learning: Concepts and Architectures.