Subgraph Pattern Neural Networks for High-Order Graph Evolution Prediction

In this work we generalize traditional node/link prediction tasks in dynamic heterogeneous networks, to consider joint prediction over larger k-node induced subgraphs. Our key insight is to incorporate the unavoidable dependencies in the training observations of induced subgraphs into both the input features and the model architecture itself via high-order dependencies. The strength of the representation is its invariance to isomorphisms and varying local neighborhood sizes, while still being able to take node/edge labels into account, and facilitating inductive reasoning (i.e., generalization to unseen portions of the network). Empirical results show that our proposed method significantly outperforms other stateof-the-art methods designed for static and/or single node/link prediction tasks. In addition, we show that our method is scalable and learns interpretable parameters. Introduction Learning predictive models of heterogeneous relational and network data is a fundamental task in machine learning and data mining (Getoor and Mihalkova 2011; Lao and Cohen 2010; Lin et al. 2015; Grover and Leskovec 2016; Nickel, Rosasco, and Poggio 2016). Much of the work in heterogeneous networks (graphs with node and edge labels) has focused on developing methods for label prediction or single link prediction. There has been relatively little development in methods that make joint predictions over larger substructures (e.g., induced k-node subgraphs). Recent research has shown rich higher-order organization of such networks (Benson, Gleich, and Leskovec 2016; Xu, Wickramarathne, and Chawla 2016) and complex subgraph evolution patterns within larger graphs (Paranjape, Benson, and Leskovec 2017). Applications range from predicting group activity on social networks (e.g., online social network ad revenues rely heavily on user activity), computational social science (e.g., predicting the dynamics of groups and their social relationships), relational learning (e.g., find missing and predicting future joint relationships in knowledge graphs). The main challenge in learning a model to predict the evolution of labeled subgraphs is to jointly account for the induced subgraph dependencies that emerge from subgraphs Copyright c © 2018, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. sharing edges. Unlike node and edge prediction tasks, it is not clear how to describe an approximate model that can account for these dependencies. A variety of recent methods have developed heuristics to encode joint label and structure information into low dimensional node or edge embeddings, but it is unclear how these ad-hoc methods can properly address the induced subgraph dependencies (Lao and Cohen 2010; Nickel, Tresp, and Kriegel 2011; Dong et al. 2014; Lin et al. 2015; Grover and Leskovec 2016; Atwood and Towsley 2016; Nickel, Rosasco, and Poggio 2016; Rahman and Al Hasan 2016). Our empirical results show that these methods tend to perform poorly in induced subgraph prediction tasks. The task of predicting induced subgraph evolution requires an approach that can take into account higher-order dependencies between the induced subgraphs (due to their shared edges and non-edges1). Our two main contributions are: (1) We target the evolution of larger graph structures than nodes and edges, which, to the best of our knowledge, has never been focused before. Traditional link prediction tasks are simpler special cases of our task. (2) We incorporate the unavoidable dependencies within the training observations of induced subgraphs into both the input features and the model architecture itself via high-order dependencies. We denote our model architecture a Subgraph Pattern Neural Network (SPNN) and show that its strength is due to a representation that is invariant to isomorphisms and varying local neighborhood sizes, can also take node/edge labels into account, and which facilitates inductive reasoning. SPNN is a discriminative feedforward neural network with hidden layers that represent the dependent subgraph patterns observed in the training data. The input features of SPNN extend the definition of induced isomorphism density (Lovász and Szegedy 2006) to a local graph neighborhood in a way that accounts for joint edges and nonedges in the induced subgraphs. Moreover, SPNN is inductive (it can be applied to unseen portions of the graph), and is isomorphic-invariant, such the learned model is invariant to node permutations. We also show that SPNN learns to predict using an interpretable neural network structure. A non-edge marks the absence of an edge The Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18)