On the Equivalence between Node Embeddings and Structural Graph Representations

This work provides the first unifying theoretical framework for node embeddings and structural graph representations, bridging methods like matrix factorization and graph neural networks. Using invariant theory, we show that the relationship between structural representations and node embeddings is analogous to that of a distribution and its samples. We prove that all tasks that can be performed by node embeddings can also be performed by structural representations and vice-versa. We also show that the concept of transductive and inductive learning is unrelated to node embeddings and graph representations, clearing another source of confusion in the literature. Finally, we introduce new practical guidelines to generating and using node embeddings, which fixes significant shortcomings of standard operating procedures used today.

[1]  Vinayak A. Rao,et al.  Relational Pooling for Graph Representations , 2019, ICML.

[2]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[3]  Martin Fürer,et al.  On the Combinatorial Power of the Weisfeiler-Lehman Algorithm , 2017, CIAC.

[4]  Ryszard S. Michalski,et al.  A theory and methodology of inductive learning , 1993 .

[5]  Risi Kondor,et al.  On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups , 2018, ICML.

[6]  Lise Getoor,et al.  Query-driven Active Surveying for Collective Classification , 2012 .

[7]  Mikhail Belkin,et al.  Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples , 2006, J. Mach. Learn. Res..

[8]  Palash Goyal,et al.  Graph Representation Ensemble Learning , 2019, 2020 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM).

[9]  Ken-ichi Kawarabayashi,et al.  What Can Neural Networks Reason About? , 2019, ICLR.

[10]  David M. Blei,et al.  Bayesian Nonparametric Poisson Factorization for Recommendation Systems , 2014, AISTATS.

[11]  C. Spearman.  General intelligence Objectively Determined and Measured , 1904 .

[12]  Alexander Gammerman,et al.  Learning by Transduction , 1998, UAI.

[13]  Samuel S. Schoenholz,et al.  Neural Message Passing for Quantum Chemistry , 2017, ICML.

[14]  Ryan L. Murphy,et al.  Janossy Pooling: Learning Deep Permutation-Invariant Functions for Variable-Size Inputs , 2018, ICLR.

[15]  Joan Bruna,et al.  On the equivalence between graph isomorphism testing and function approximation with GNNs , 2019, NeurIPS.

[16]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[17]  Qiongkai Xu,et al.  GraRep: Learning Graph Representations with Global Structural Information , 2015, CIKM.

[18]  Sanjeev Arora,et al.  A Latent Variable Model Approach to PMI-based Word Embeddings , 2015, TACL.

[19]  Max Welling,et al.  Modeling Relational Data with Graph Convolutional Networks , 2017, ESWC.

[20]  Martin Grohe,et al.  Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networks , 2018, AAAI.

[21]  Michael A. Osborne,et al.  On the Limitations of Representing Functions on Sets , 2019, ICML.

[22]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[23]  Lise Getoor,et al.  Collective Classification in Network Data , 2008, AI Mag..

[24]  Ruslan Salakhutdinov,et al.  Revisiting Semi-Supervised Learning with Graph Embeddings , 2016, ICML.

[25]  Emmanuel J. Candès,et al.  Exact Matrix Completion via Convex Optimization , 2009, Found. Comput. Math..

[26]  Alexander J. Smola,et al.  Distributed large-scale natural graph factorization , 2013, WWW.

[27]  Razvan Pascanu,et al.  Relational inductive biases, deep learning, and graph networks , 2018, ArXiv.

[28]  Xin-She Yang,et al.  Introduction to Algorithms , 2021, Nature-Inspired Optimization Algorithms.

[29]  Robert D. Kleinberg.  Geographic Routing Using Hyperbolic Space , 2007, IEEE INFOCOM 2007 - 26th IEEE International Conference on Computer Communications.

[30]  Matthew D. Hoffman,et al.  Variational Autoencoders for Collaborative Filtering , 2018, WWW.

[31]  Tim Austin.  On exchangeable random variables and the statistics of large graphs and hypergraphs , 2008, 0801.1698.

[32]  Jure Leskovec,et al.  Position-aware Graph Neural Networks , 2019, ICML.

[33]  Stefano Ermon,et al.  Graphite: Iterative Generative Modeling of Graphs , 2018, ICML.

[34]  Nicholas Ruozzi,et al.  Correlated Variational Auto-Encoders , 2019, ICML.

[35]  Jure Leskovec,et al.  Predicting multicellular function through multi-layer tissue networks , 2017, Bioinform..

[36]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[37]  Max Welling,et al.  Variational Graph Auto-Encoders , 2016, ArXiv.

[38]  Arthur Gretton,et al.  Parallel Gibbs Sampling: From Colored Fields to Thin Junction Trees , 2011, AISTATS.

[39]  David M. Blei,et al.  Content-based recommendations with Poisson factorization , 2014, NIPS.

[40]  Ah Chung Tsoi,et al.  The Graph Neural Network Model , 2009, IEEE Transactions on Neural Networks.

[41]  J. Pearl,et al.  An Axiomatic Characterization of Causal Counterfactuals , 1998 .

[42]  Roger Allan Cropp,et al.  Construction of a trophically complex near-shore Antarctic food web model using the Conservative Normal framework with structural coexistence , 2015 .

[43]  Max Welling,et al.  Graph Convolutional Matrix Completion , 2017, ArXiv.

[44]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[45]  Jure Leskovec,et al.  Learning Structural Node Embeddings via Diffusion Wavelets , 2017, KDD.

[46]  Thorsten Joachims,et al.  Playlist prediction via metric embedding , 2012, KDD.

[47]  Jeffrey Wood,et al.  A unifying framework for invariant pattern recognition , 1996, Pattern Recognit. Lett..

[48]  Alessandro Epasto,et al.  Is a Single Embedding Enough? Learning Node Representations that Capture Multiple Social Contexts , 2019, WWW.

[49]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[50]  N. Stenseth,et al.  Population regulation in snowshoe hare and Canadian lynx: asymmetric food web configurations between hare and lynx. , 1997, Proceedings of the National Academy of Sciences of the United States of America.

[51]  Alon Orlitsky,et al.  On Learning Distributions from their Samples , 2015, COLT.

[52]  Zoubin Ghahramani,et al.  Combining active learning and semi-supervised learning using Gaussian fields and harmonic functions , 2003, ICML 2003.

[53]  Bernhard Schölkopf,et al.  Learning with Local and Global Consistency , 2003, NIPS.

[54]  Ittai Abraham,et al.  Advances in metric embedding theory , 2006, STOC '06.

[55]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[56]  Ran Raz,et al.  Lower Bounds on the Distortion of Embedding Finite Metric Spaces in Graphs , 1998, Discret. Comput. Geom..

[57]  Steven Skiena,et al.  DeepWalk: online learning of social representations , 2014, KDD.

[58]  Danai Koutra,et al.  From Community to Role-based Graph Embeddings , 2019, ArXiv.

[59]  Alán Aspuru-Guzik,et al.  Convolutional Networks on Graphs for Learning Molecular Fingerprints , 2015, NIPS.

[60]  Xavier Bresson,et al.  Geometric Matrix Completion with Recurrent Multi-Graph Neural Networks , 2017, NIPS.

[61]  Jure Leskovec,et al.  node2vec: Scalable Feature Learning for Networks , 2016, KDD.

[62]  Thomas L. Griffiths,et al.  Evaluating Vector-Space Models of Word Representation, or, The Unreasonable Effectiveness of Counting Words Near Other Words , 2017, CogSci.

[63]  Daniel R. Figueiredo,et al.  struc2vec: Learning Node Representations from Structural Identity , 2017, KDD.

[64]  R. Graham,et al.  On isometric embeddings of graphs , 1985 .

[65]  Yixin Chen,et al.  Link Prediction Based on Graph Neural Networks , 2018, NeurIPS.

[66]  Jure Leskovec,et al.  How Powerful are Graph Neural Networks? , 2018, ICLR.

[67]  Yaron Lipman,et al.  Invariant and Equivariant Graph Networks , 2018, ICLR.

[68]  Bert Huang,et al.  Learning a Distance Metric from a Network , 2011, NIPS.

[69]  Pablo A. Parrilo,et al.  Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization , 2007, SIAM Rev..

[70]  Jian Pei,et al.  Asymmetric Transitivity Preserving Graph Embedding , 2016, KDD.

[71]  Bert Huang,et al.  Recurrent collective classification , 2017, Knowledge and Information Systems.

[72]  O. Kallenberg.  Foundations of Modern Probability , 2021, Probability Theory and Stochastic Modelling.

[73]  J. Bourgain.  On lipschitz embedding of finite metric spaces in Hilbert space , 1985 .

[74]  Zhiyuan Liu,et al.  Graph Neural Networks: A Review of Methods and Applications , 2018, AI Open.

[75]  Ruslan Salakhutdinov,et al.  Probabilistic Matrix Factorization , 2007, NIPS.

[76]  Jure Leskovec,et al.  Representation Learning on Graphs: Methods and Applications , 2017, IEEE Data Eng. Bull..

[77]  Omer Levy,et al.  Neural Word Embedding as Implicit Matrix Factorization , 2014, NIPS.

[78]  Philip S. Yu,et al.  A Comprehensive Survey on Graph Neural Networks , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[79]  Yee Whye Teh,et al.  Probabilistic symmetry and invariant neural networks , 2019, J. Mach. Learn. Res..

[80]  Danai Koutra,et al.  RolX: structural role extraction & mining in large graphs , 2012, KDD.

[81]  Mikhail Belkin,et al.  Laplacian Eigenmaps and Spectral Techniques for Embedding and Clustering , 2001, NIPS.

[82]  Joseph Y. Halpern.  Axiomatizing Causal Reasoning , 1998, UAI.