Large Scale Learning on Non-Homophilous Graphs: New Benchmarks and Strong Simple Methods

Many widely used datasets for graph machine learning tasks have generally been homophilous, where nodes with similar labels connect to each other. Recently, new Graph Neural Networks (GNNs) have been developed that move beyond the homophily regime; however, their evaluation has often been conducted on small graphs with limited application domains. We collect and introduce diverse nonhomophilous datasets from a variety of application areas that have up to 384x more nodes and 1398x more edges than prior datasets. We further show that existing scalable graph learning and graph minibatching techniques lead to performance degradation on these non-homophilous datasets, thus highlighting the need for further work on scalable non-homophilous methods. To address these concerns, we introduce LINKX — a strong simple method that admits straightforward minibatch training and inference. Extensive experimental results with representative simple methods and GNNs across our proposed datasets show that LINKX achieves state-ofthe-art performance for learning on non-homophilous graphs. Our codes and data are available at https://github.com/CUAI/Non-Homophily-Large-Scale.

[1]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[2]  Jiliang Tang,et al.  Node Similarity Preserving Graph Convolutional Networks , 2020, WSDM.

[3]  Yuxiao Dong,et al.  Microsoft Academic Graph: When experts are not enough , 2020, Quantitative Science Studies.

[4]  Donald F. Towsley,et al.  Resisting structural re-identification in anonymized social networks , 2010, The VLDB Journal.

[5]  Derek Lim,et al.  Expertise and Dynamics within Crowdsourced Musical Knowledge Curation: A Case Study of the Genius Platform , 2020, ArXiv.

[6]  J. Leskovec,et al.  Open Graph Benchmark: Datasets for Machine Learning on Graphs , 2020, NeurIPS.

[7]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[8]  M. Newman,et al.  Mixing patterns in networks. , 2002, Physical review. E, Statistical, nonlinear, and soft matter physics.

[9]  Kristen M. Altenburger,et al.  Monophily in social networks introduces similarity among friends-of-friends , 2018, Nature Human Behaviour.

[10]  Lise Getoor,et al.  Collective Classification in Network Data , 2008, AI Mag..

[11]  Jure Leskovec,et al.  {SNAP Datasets}: {Stanford} Large Network Dataset Collection , 2014 .

[12]  Jimeng Sun,et al.  Social influence analysis in large-scale networks , 2009, KDD.

[13]  Yongtian Wang,et al.  Deep Surface Normal Estimation With Hierarchical RGB-D Fusion , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Stephan Günnemann,et al.  Pitfalls of Graph Neural Network Evaluation , 2018, ArXiv.

[15]  Jure Leskovec,et al.  Graph Convolutional Neural Networks for Web-Scale Recommender Systems , 2018, KDD.

[16]  Cynthia Dwork,et al.  Wherefore art thou r3579x?: anonymized social networks, hidden patterns, and structural steganography , 2007, WWW '07.

[17]  Austin R. Benson,et al.  Higher-order Homophily is Combinatorially Impossible , 2021, ArXiv.

[18]  P. Erdos,et al.  On the evolution of random graphs , 1984 .

[19]  Johan Ugander,et al.  Decoupled Smoothing on Graphs , 2019, WWW.

[20]  Alice H. Oh,et al.  How to Find Your Friendly Neighborhood: Graph Attention Design with Self-Supervision , 2022, ICLR.

[21]  Frank Hutter,et al.  Decoupled Weight Decay Regularization , 2017, ICLR.

[22]  Takanori Maehara,et al.  Revisiting Graph Neural Networks: All We Have is Low-Pass Filters , 2019, ArXiv.

[23]  Xiao Wang,et al.  Beyond Low-frequency Information in Graph Convolutional Networks , 2021, AAAI.

[24]  Woodrow Hartzog,et al.  The Case for Online Obscurity , 2012 .

[25]  Udi Weinsberg,et al.  Friend or Faux: Graph-Based Early Detection of Fake Accounts on Social Networks , 2020, WWW.

[26]  Leto Peel,et al.  Graph-based semi-supervised learning for relational networks , 2016, SDM.

[27]  Cao Xiao,et al.  FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling , 2018, ICLR.

[28]  Danai Koutra,et al.  Graph Neural Networks with Heterophily , 2021, AAAI.

[29]  Stephan Günnemann,et al.  Predict then Propagate: Graph Neural Networks meet Personalized PageRank , 2018, ICLR.

[30]  L. Akoglu,et al.  Beyond Homophily in Graph Neural Networks: Current Limitations and Effective Designs , 2020, NeurIPS.

[31]  Qian Huang,et al.  Combining Label Propagation and Simple Models Out-performs Graph Neural Networks , 2020, ICLR.

[32]  Olgica Milenkovic,et al.  Adaptive Universal Generalized PageRank Graph Neural Network , 2021, ICLR.

[33]  Sepp Hochreiter,et al.  Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) , 2015, ICLR.

[34]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[35]  Vitaly Shmatikov,et al.  Robust De-anonymization of Large Sparse Datasets , 2008, 2008 IEEE Symposium on Security and Privacy (sp 2008).

[36]  Christos Faloutsos,et al.  Detecting Fraudulent Personalities in Networks of Online Auctioneers , 2006, PKDD.

[37]  Johannes Klicpera,et al.  Scaling Graph Neural Networks with Approximate PageRank , 2020, KDD.

[38]  Vitaly Shmatikov,et al.  De-anonymizing Social Networks , 2009, 2009 30th IEEE Symposium on Security and Privacy.

[39]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[40]  Rajgopal Kannan,et al.  GraphSAINT: Graph Sampling Based Inductive Learning Method , 2019, ICLR.

[41]  Timnit Gebru,et al.  Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification , 2018, FAT.

[42]  Bernhard Schölkopf,et al.  Learning with Local and Global Consistency , 2003, NIPS.

[43]  Wolfgang Gatterbauer,et al.  Semi-Supervised Learning with Heterophily , 2014, ArXiv.

[44]  Kristina Lerman,et al.  MixHop: Higher-Order Graph Convolutional Architectures via Sparsified Neighborhood Mixing , 2019, ICML.

[45]  Mason A. Porter,et al.  Social Structure of Facebook Networks , 2011, ArXiv.

[46]  Lise Getoor,et al.  To join or not to join: the illusion of privacy in social networks with mixed public and private user profiles , 2009, WWW '09.

[47]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[48]  Samy Bengio,et al.  Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks , 2019, KDD.

[49]  Takuya Akiba,et al.  Optuna: A Next-generation Hyperparameter Optimization Framework , 2019, KDD.

[50]  Kevin Chen-Chuan Chang,et al.  Geom-GCN: Geometric Graph Convolutional Networks , 2020, ICLR.

[51]  Rik Sarkar,et al.  Twitch Gamers: a Dataset for Evaluating Proximity Preserving and Structural Role-based Node Embeddings , 2021, ArXiv.

[52]  Christoph Zetzsche,et al.  Early vs Late Fusion in Multimodal Convolutional Neural Networks , 2020, 2020 IEEE 23rd International Conference on Information Fusion (FUSION).

[53]  M. McPherson,et al.  Birds of a Feather: Homophily in Social Networks , 2001 .

[54]  Dallas Card,et al.  The Values Encoded in Machine Learning Research , 2021, ArXiv.

[55]  Xavier Bresson,et al.  Benchmarking Graph Neural Networks , 2020, ArXiv.

[56]  Christos Faloutsos,et al.  Netprobe: a fast and scalable system for fraud detection in online auction networks , 2007, WWW '07.

[57]  Christos Faloutsos,et al.  Graphs over time: densification laws, shrinking diameters and possible explanations , 2005, KDD '05.

[58]  Daniel J. Singer.  To the Best of Our Knowledge , 2021, The Philosophical Review.

[59]  Paul Honeine,et al.  Analyzing the Expressive Power of Graph Neural Networks in a Spectral Perspective , 2021, ICLR.

[60]  Rik Sarkar,et al.  Multi-scale Attributed Node Embedding , 2019, J. Complex Networks.

[61]  Yaliang Li,et al.  Simple and Deep Graph Convolutional Networks , 2020, ICML.

[62]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[63]  Kilian Q. Weinberger,et al.  Simplifying Graph Convolutional Networks , 2019, ICML.

[64]  N. Sloane.  The on-line encyclopedia of integer sequences , 2018, Notices of the American Mathematical Society.

[65]  Razvan Pascanu,et al.  Relational inductive biases, deep learning, and graph networks , 2018, ArXiv.

[66]  Viktor K. Prasanna,et al.  Accurate, Efficient and Scalable Graph Embedding , 2018, 2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS).

[67]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[68]  Pietro Liò,et al.  Graph Attention Networks , 2017, ICLR.

[69]  Zhiru Zhang,et al.  GraphZoom: A multi-level spectral approach for accurate and scalable graph embedding , 2019, ICLR.

[70]  Jan Eric Lenssen,et al.  Fast Graph Representation Learning with PyTorch Geometric , 2019, ArXiv.

[71]  Danai Koutra,et al.  Two Sides of the Same Coin: Heterophily and Oversmoothing in Graph Convolutional Neural Networks , 2021, ArXiv.

[72]  Kristina Lerman,et al.  A Survey on Bias and Fairness in Machine Learning , 2019, ACM Comput. Surv..

[73]  Ruslan Salakhutdinov,et al.  Revisiting Semi-Supervised Learning with Graph Embeddings , 2016, ICML.

[74]  Austin R. Benson,et al.  Residual Correlation in Graph Neural Network Regression , 2020, KDD.

[75]  Tsuyoshi Murata,et al.  Stacked Graph Filter , 2020, ArXiv.

[76]  Ken-ichi Kawarabayashi,et al.  Representation Learning on Graphs with Jumping Knowledge Networks , 2018, ICML.

[77]  L. Takac.  DATA ANALYSIS IN PUBLIC SOCIAL NETWORKS , 2012 .

[78]  W. Kittisupamongkol.  Two sides of the same coin? , 2010, Singapore medical journal.

[79]  N. J. A. Sloane,et al.  The On-Line Encyclopedia of Integer Sequences , 2003, Electron. J. Comb..