A Neural Turing~Machine for Conditional Transition Graph Modeling

Graphs are an essential part of many machine learning problems such as analysis of parse trees, social networks, knowledge graphs, transportation systems, and molecular structures. Applying machine learning in these areas typically involves learning the graph structure and the relationship between the nodes of the graph. However, learning the graph structure is often complex, particularly when the graph is cyclic, and the transitions from one node to another are conditioned such as graphs used to represent a finite state machine. To solve this problem, we propose to extend the memory based Neural Turing Machine (NTM) with two novel additions. We allow for transitions between nodes to be influenced by information received from external environments, and we let the NTM learn the context of those transitions. We refer to this extension as the Conditional Neural Turing Machine (CNTM). We show that the CNTM can infer conditional transition graphs by empirically verifiying the model on two data sets: a large set of randomly generated graphs, and a graph modeling the information retrieval process during certain crisis situations. The results show that the CNTM is able to reproduce the paths inside the graph with accuracy ranging from 82,12% for 10 nodes graphs to 65,25% for 100 nodes graphs.

[1]  Geoffrey E. Hinton.  Tensor Product Variable Binding and the Representation of Symbolic Structures in Connectionist Systems , 1991 .

[2]  Mohammad Al Hasan,et al.  Link prediction using supervised learning , 2006 .

[3]  Jordan B. Pollack,et al.  Recursive Distributed Representations , 1990, Artif. Intell..

[4]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[5]  J. Fodor,et al.  Connectionism and cognitive architecture: A critical analysis , 1988, Cognition.

[6]  Yoshua Bengio,et al.  A Neural Knowledge Language Model , 2016, ArXiv.

[7]  Mohammad Al Hasan,et al.  A Survey of Link Prediction in Social Networks , 2011, Social Network Data Analytics.

[8]  Sergio Gomez Colmenarejo,et al.  Hybrid computing using a neural network with dynamic external memory , 2016, Nature.

[9]  Tony A. Plate,et al.  Holographic reduced representations , 1995, IEEE Trans. Neural Networks.

[10]  David Liben-Nowell,et al.  The link-prediction problem for social networks , 2007 .

[11]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[12]  Rakesh Gupta,et al.  Knowledge Representation and Bayesian Inference for Response to Situations , 2005 .

[13]  Andreas Prinz,et al.  Identifying First Responders Information Needs: Supporting Search and Rescue Operations for Fire Emergency Response , 2016, Int. J. Inf. Syst. Crisis Response Manag..

[14]  Yin Zhang,et al.  Scalable proximity estimation and link prediction in online social networks , 2009, IMC '09.

[15]  Geoffrey E. Hinton,et al.  Learning distributed representations of concepts. , 1989 .

[16]  Geoffrey E. Hinton.  BoltzCONS: Dynamic Symbol Structures in a Connectionist Network , 1991 .

[17]  Shiu-li Huang,et al.  Designing utility-based recommender systems for e-commerce: Evaluation of preference-elicitation methods , 2011, Electron. Commer. Res. Appl..

[18]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[19]  Alex Graves,et al.  Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes , 2016, NIPS.

[20]  Navdeep Jaitly,et al.  Pointer Networks , 2015, NIPS.

[21]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[22]  Alex Graves,et al.  Neural Turing Machines , 2014, ArXiv.

[23]  E. Xing,et al.  Mixed Membership Stochastic Block Models for Relational Data with Application to Protein-Protein Interactions , 2006 .