Isometric Graph Neural Networks

Many tasks that rely on representations of nodes in graphs would benefit if those representations were faithful to distances between nodes in the graph. Geometric techniques to extract such representations have poor scaling over large graph size, and recent advances in Graph Neural Network (GNN) algorithms have limited ability to reflect graph distance information beyond the first degree neighborhood. To enable this highly desired capability, we propose a technique to learn Isometric Graph Neural Networks (IGNN), which requires changing the input representation space and loss function to enable any GNN algorithm to generate representations that reflect distances between nodes. We experiment with the isometric technique on several GNN architectures for modeling multiple prediction tasks on multiple datasets. In addition to an improvement in AUC-ROC as high as $43\%$ in these experiments, we observe a consistent and substantial improvement as high as 400% in Kendall's Tau (KT), a measure that directly reflects distance information, demonstrating that the learned embeddings do account for graph distances.

[1]  D. Watts.  Networks, Dynamics, and the Small‐World Phenomenon1 , 1999, American Journal of Sociology.

[2]  Razvan Pascanu,et al.  Vector-based navigation using grid-like representations in artificial agents , 2018, Nature.

[3]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[4]  Silvio Savarese,et al.  Social LSTM: Human Trajectory Prediction in Crowded Spaces , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Yunhai Tong,et al.  A Position Encoding Convolutional Neural Network Based on Dependency Tree for Relation Classification , 2016, EMNLP.

[6]  Jian Pei,et al.  Asymmetric Transitivity Preserving Graph Embedding , 2016, KDD.

[7]  Xiaogang Wang,et al.  Spatial As Deep: Spatial CNN for Traffic Scene Understanding , 2017, AAAI.

[8]  Jure Leskovec,et al.  Predicting multicellular function through multi-layer tissue networks , 2017, Bioinform..

[9]  Song-Chun Zhu,et al.  Learning Grid Cells as Vector Representation of Self-Position Coupled with Matrix Representation of Self-Motion , 2018, ICLR.

[10]  Sen Jia,et al.  How Much Position Information Do Convolutional Neural Networks Encode? , 2020, ICLR.

[11]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[12]  J. Bourgain.  On lipschitz embedding of finite metric spaces in Hilbert space , 1985 .

[13]  W. B. Johnson,et al.  Extensions of Lipschitz mappings into Hilbert space , 1984 .

[14]  R. Forthofer,et al.  Rank Correlation Methods , 1981 .

[15]  Jure Leskovec,et al.  node2vec: Scalable Feature Learning for Networks , 2016, KDD.

[16]  W. Knight.  A Computer Method for Calculating Kendall's Tau with Ungrouped Data , 1966 .

[17]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[18]  Balasubramaniam Srinivasan,et al.  On the Equivalence between Node Embeddings and Structural Graph Representations , 2019, ICLR 2020.

[19]  Christos Faloutsos,et al.  Graph evolution: Densification and shrinking diameters , 2006, TKDD.

[20]  Monika Henzinger,et al.  Dynamic Approximate All-Pairs Shortest Paths: Breaking the O(mn) Barrier and Derandomization , 2013, 2013 IEEE 54th Annual Symposium on Foundations of Computer Science.

[21]  Jure Leskovec,et al.  Position-aware Graph Neural Networks , 2019, ICML.

[22]  Kishan G. Mehrotra,et al.  A fast sorting algorithm for aptamer identification using deep sequencing , 2014, 2014 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM 2014).

[23]  Joan Bruna,et al.  On the equivalence between graph isomorphism testing and function approximation with GNNs , 2019, NeurIPS.

[24]  J. Nash.  The imbedding problem for Riemannian manifolds , 1956 .

[25]  Jure Leskovec,et al.  Representation Learning on Graphs: Methods and Applications , 2017, IEEE Data Eng. Bull..

[26]  A Time-Aware Inductive Representation Learning Strategy for Heterogeneous Graphs , 2019 .

[27]  Alexander J. Smola,et al.  Distributed large-scale natural graph factorization , 2013, WWW.

[28]  Razvan Pascanu,et al.  Relational inductive biases, deep learning, and graph networks , 2018, ArXiv.

[29]  Krzysztof Janowicz,et al.  Multi-Scale Representation Learning for Spatial Feature Distributions using Grid Cells , 2020, ICLR.

[30]  Jun Zhao,et al.  Relation Classification via Convolutional Deep Neural Network , 2014, COLING.

[31]  Steven Skiena,et al.  DeepWalk: online learning of social representations , 2014, KDD.

[32]  Pinar Yanardag,et al.  Deep Graph Kernels , 2015, KDD.

[33]  Jure Leskovec,et al.  How Powerful are Graph Neural Networks? , 2018, ICLR.

[34]  Sanja Fidler,et al.  Order-Embeddings of Images and Language , 2015, ICLR.

[35]  Douwe Kiela,et al.  Hyperbolic Graph Neural Networks , 2019, NeurIPS.

[36]  Karsten M. Borgwardt,et al.  Halting in Random Walk Kernels , 2015, NIPS.

[37]  Renjie Liao,et al.  Efficient Graph Generation with Graph Recurrent Attention Networks , 2019, NeurIPS.

[38]  Xue-Xin Wei,et al.  Emergence of grid-like representations by training recurrent neural networks to perform spatial localization , 2018, ICLR.

[39]  Mikhail Belkin,et al.  Laplacian Eigenmaps and Spectral Techniques for Embedding and Clustering , 2001, NIPS.

[40]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[41]  Raia Hadsell,et al.  Neural Execution of Graph Algorithms , 2020, ICLR.

[42]  Jure Leskovec,et al.  Hyperbolic Graph Convolutional Neural Networks , 2019, NeurIPS.

[43]  Pietro Liò,et al.  Graph Attention Networks , 2017, ICLR.

[44]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[45]  Gert R. G. Lanckriet,et al.  Partial order embedding with multiple kernels , 2009, ICML '09.

[46]  Peter Frankl,et al.  The Johnson-Lindenstrauss lemma and the sphericity of some graphs , 1987, J. Comb. Theory B.

[47]  Jan Eric Lenssen,et al.  Fast Graph Representation Learning with PyTorch Geometric , 2019, ArXiv.

[48]  Samuel S. Schoenholz,et al.  Neural Message Passing for Quantum Chemistry , 2017, ICML.

[49]  Yaron Lipman,et al.  Provably Powerful Graph Networks , 2019, NeurIPS.