Learning to Make Predictions on Graphs with Autoencoders

We examine two fundamental tasks associated with graph representation learning: link prediction and semi-supervised node classification. We present a novel autoencoder architecture capable of learning a joint representation of both local graph structure and available node features for the multi-task learning of link prediction and node classification. Our autoencoder architecture is efficiently trained end-to-end in a single learning stage to simultaneously perform link prediction and node classification, whereas previous related methods require multiple training steps that are difficult to optimize. We provide a comprehensive empirical evaluation of our models on nine benchmark graph-structured datasets and demonstrate significant improvement over related methods for graph representation learning. Reference code and data are available at https://github.com/vuptran/graph-representation-learning.

[1]  Rich Caruana,et al.  Multitask Learning: A Knowledge-Based Source of Inductive Bias , 1993, ICML.

[2]  Mingzhe Wang,et al.  LINE: Large-scale Information Network Embedding , 2015, WWW.

[3]  Florian Strub,et al.  Hybrid Collaborative Filtering with Autoencoders , 2016 .

[4]  Zan Huang,et al.  The Time-Series Link Prediction Problem with Applications in Communication Surveillance , 2009, INFORMS J. Comput..

[5]  L. Getoor,et al.  Link-Based Classification , 2003, Encyclopedia of Machine Learning and Data Mining.

[6]  Santanu Chaudhury,et al.  A Data and Model-Parallel, Distributed and Scalable Framework for Training of Deep Networks in Apache Spark , 2017, ArXiv.

[7]  Jure Leskovec,et al.  Representation Learning on Graphs: Methods and Applications , 2017, IEEE Data Eng. Bull..

[8]  Hassan Khosravi,et al.  A Survey on Statistical Relational Learning , 2010, Canadian Conference on AI.

[9]  Phi Vu Tran,et al.  A Fully Convolutional Neural Network for Cardiac Segmentation in Short-Axis MRI , 2016, ArXiv.

[10]  Francesco Bonchi,et al.  Cold start link prediction , 2010, KDD.

[11]  Yongxin Yang,et al.  Trace Norm Regularised Deep Multi-Task Learning , 2016, ICLR.

[12]  David W. Aha,et al.  Transforming Graph Data for Statistical Relational Learning , 2012, J. Artif. Intell. Res..

[13]  Lise Getoor,et al.  Entity and Relationship Labeling in Affiliation Networks , 2006 .

[14]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[15]  Peng Wang,et al.  Link prediction in social networks: the state-of-the-art , 2014, Science China Information Sciences.

[16]  GetoorLise,et al.  Network-based drug-target interaction prediction with probabilistic soft logic , 2014 .

[17]  Weixiong Zhang,et al.  A Marginalized Denoising Method for Link Prediction in Relational Data , 2014, SDM.

[18]  Huan Liu,et al.  Leveraging social media networks for classification , 2011, Data Mining and Knowledge Discovery.

[19]  Wenwu Zhu,et al.  Structural Deep Network Embedding , 2016, KDD.

[20]  Charles Elkan,et al.  Link Prediction via Matrix Factorization , 2011, ECML/PKDD.

[21]  Guillaume Gravier,et al.  Bidirectional Joint Representation Learning with Symmetrical Deep Neural Networks for Multimodal and Crossmodal Applications , 2016, ICMR.

[22]  Max Welling,et al.  Variational Graph Auto-Encoders , 2016, ArXiv.

[23]  Scott Sanner,et al.  AutoRec: Autoencoders Meet Collaborative Filtering , 2015, WWW.

[24]  Boris Ginsburg,et al.  Training Deep AutoEncoders for Collaborative Filtering , 2017, ArXiv.

[25]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[26]  Yehuda Koren,et al.  Matrix Factorization Techniques for Recommender Systems , 2009, Computer.

[27]  Vikas Joshi,et al.  Modified Mean and Variance Normalization: Transforming to Utterance-Specific Estimates , 2016, Circuits Syst. Signal Process..

[28]  Jianfeng Gao,et al.  Embedding Entities and Relations for Learning and Inference in Knowledge Bases , 2014, ICLR.

[29]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[30]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[31]  James R. Foulds,et al.  Collective Spammer Detection in Evolving Multi-Relational Social Networks , 2015, KDD.

[32]  Lise Getoor,et al.  Collective Classification in Network Data , 2008, AI Mag..

[33]  Ruslan Salakhutdinov,et al.  Revisiting Semi-Supervised Learning with Graph Embeddings , 2016, ICML.

[34]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[35]  Steven Skiena,et al.  DeepWalk: online learning of social representations , 2014, KDD.

[36]  Ross B. Girshick,et al.  Focal Loss for Dense Object Detection , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.