Hybrid computing using a neural network with dynamic external memory

Artificial neural networks are remarkably adept at sensory processing, sequence learning and reinforcement learning, but are limited in their ability to represent variables and data structures and to store data over long timescales, owing to the lack of an external memory. Here we introduce a machine learning model called a differentiable neural computer (DNC), which consists of a neural network that can read from and write to an external memory matrix, analogous to the random-access memory in a conventional computer. Like a conventional computer, it can use its memory to represent and manipulate complex data structures, but, like a neural network, it can learn to do so from data. When trained with supervised learning, we demonstrate that a DNC can successfully answer synthetic questions designed to emulate reasoning and inference problems in natural language. We show that it can learn tasks such as finding the shortest path between specified points and inferring the missing links in randomly generated graphs, and then generalize these tasks to specific graphs such as transport networks and family trees. When trained with reinforcement learning, a DNC can complete a moving blocks puzzle in which changing goals are specified by sequences of symbols. Taken together, our results demonstrate that DNCs have the capacity to solve complex, structured tasks that are inaccessible to neural networks without external read–write memory.

[1]  Terry Winograd,et al.  Procedures As A Representation For Data In A Computer Program For Understanding Natural Language , 1971 .

[2]  B. Skinner,et al.  Symbolic Communication Between Two Pigeons, (Columba livia domestica) , 1980, Science.

[3]  Douglas L. Hintzman MINERVA 2: A simulation model of human memory , 1984 .

[4]  Pentti Kanerva,et al.  Sparse Distributed Memory , 1988 .

[5]  Shun-ichi Amari,et al.  Characteristics of sparsely encoded associative memory , 1989, Neural Networks.

[6]  Geoffrey E. Hinton Learning distributed representations of concepts. , 1989 .

[7]  Paul J. Werbos,et al.  Backpropagation Through Time: What It Does and How to Do It , 1990 .

[8]  James L. McClelland,et al.  Hippocampal conjunctive encoding, storage, and recall: Avoiding a trade‐off , 1994, Hippocampus.

[9]  James L. McClelland,et al.  Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. , 1995, Psychological review.

[10]  Paul R. Wilson,et al.  Dynamic Storage Allocation: A Survey and Critical Review , 1995, IWMM.

[11]  James L. McClelland,et al.  Considerations arising from a complementary learning systems perspective on hippocampus and neocortex , 1996, Hippocampus.

[12]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[13]  D. Johnston,et al.  A Synaptically Controlled, Associative Signal for Hebbian Plasticity in Hippocampal Neurons , 1997, Science.

[14]  Yishay Mansour,et al.  Policy Gradient Methods for Reinforcement Learning with Function Approximation , 1999, NIPS.

[15]  G. Marcus The Algebraic Mind: Integrating Connectionism and Cognitive Science , 2001 .

[16]  Marc W. Howard,et al.  A distributed representation of temporal context , 2002 .

[17]  Ronald J. Williams Simple statistical gradient-following algorithms for connectionist reinforcement learning , 2004, Machine Learning.

[18]  L. Abbott,et al.  Cascade Models of Synaptically Stored Memories , 2005, Neuron.

[19]  A. Torralba,et al.  The role of context in object recognition , 2007, Trends in Cognitive Sciences.

[20]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[21]  Surya Ganguli,et al.  Memory traces in dynamical systems , 2008, Proceedings of the National Academy of Sciences.

[22]  John Langford,et al.  Search-based structured prediction , 2009, Machine Learning.

[23]  Jason Weston,et al.  Curriculum learning , 2009, ICML '09.

[24]  Geoffrey J. Gordon,et al.  A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning , 2010, AISTATS.

[25]  Marc'Aurelio Ranzato,et al.  Large Scale Distributed Deep Networks , 2012, NIPS.

[26]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[27]  Geoffrey E. Hinton,et al.  Speech recognition with deep recurrent neural networks , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[28]  Jonathan D. Cohen,et al.  Indirection and symbol-like processing in the prefrontal cortex and basal ganglia , 2013, Proceedings of the National Academy of Sciences.

[29]  Alex Graves,et al.  Generating Sequences With Recurrent Neural Networks , 2013, ArXiv.

[30]  Léon Bottou,et al.  From machine learning to machine reasoning , 2011, Machine Learning.

[31]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[32]  Jason Weston,et al.  End-To-End Memory Networks , 2015, NIPS.

[33]  Jason Weston,et al.  Memory Networks , 2014, ICLR.

[34]  Joshua B. Tenenbaum,et al.  Human-level concept learning through probabilistic program induction , 2015, Science.

[35]  Navdeep Jaitly,et al.  Pointer Networks , 2015, NIPS.

[36]  Alex Graves,et al.  DRAW: A Recurrent Neural Network For Image Generation , 2015, ICML.

[37]  Phil Blunsom,et al.  Teaching Machines to Read and Comprehend , 2015, NIPS.

[38]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[39]  Daan Wierstra,et al.  One-Shot Generalization in Deep Generative Models , 2016, ICML.

[40]  ST Johnston,et al.  Paradox of pattern separation and adult neurogenesis: A dual role for new neurons balancing memory resolution and robustness , 2016, Neurobiology of Learning and Memory.

[41]  Daan Wierstra,et al.  Meta-Learning with Memory-Augmented Neural Networks , 2016, ICML.

[42]  James L. McClelland,et al.  What Learning Systems do Intelligent Agents Need? Complementary Learning Systems Theory Updated , 2016, Trends in Cognitive Sciences.