Generalization by symbolic abstraction in cascaded recurrent networks

Generalization performance in recurrent neural networks is enhanced by cascading several networks. By discretizing abstractions induced in one network, other networks can operate on a coarse symbolic level with increased performance on sparse and structural prediction tasks. The level of systematicity exhibited by the cascade of recurrent networks is assessed on the basis of three language domains.

[1]  Mikael Bodén,et al.  Learning the Dynamics of Embedded Clauses , 2003, Applied Intelligence.

[2]  Dirk V. Arnold,et al.  Syntactic systematicity arising from semantic predictions in a Hebbian-competitive network , 2001, Connect. Sci..

[3]  Gary F Marcus,et al.  Language acquisition in the absence of explicit negative evidence: can simple recurrent networks obviate the need for domain-specific learning devices? , 1999, Cognition.

[4]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[5]  Steven Phillips,et al.  Constituent similarity and systematicity: The limits of first-order connectionism , 2000, Connect. Sci..

[6]  Douglas L. T. Rohde,et al.  Simple recurrent networks can distinguish non-occurring from ungrammatical sentences given appropriate task structure: reply to Marcus , 1999, Cognition.

[7]  J. Fodor,et al.  Connectionism and cognitive architecture: A critical analysis , 1988, Cognition.

[8]  Jürgen Schmidhuber,et al.  Discovering Predictable Classifications , 1993, Neural Computation.

[9]  Stefano Nolfi,et al.  Extracting Regularities in Space and Time Through a Cascade of Prediction Networks: The Case of a Mobile Robot Navigating in a Structured Environment , 1999, Connect. Sci..

[10]  Janet Wiles,et al.  Context-free and context-sensitive dynamics in recurrent neural networks , 2000, Connect. Sci..

[11]  J. Pollack.  The Induction of Dynamical Recognizers , 1996, Machine Learning.

[12]  Robert F. Hadley.  Systematicity in Connectionist Language Learning , 1994 .

[13]  Nick Chater,et al.  Toward a connectionist model of recursion in human linguistic performance , 1999 .

[14]  Zenon W. Pylyshyn,et al.  Connectionism and cognitive architecture , 1993 .

[15]  Paul Rodríguez,et al.  A Recurrent Neural Network that Learns to Count , 1999, Connect. Sci..

[16]  Douglas L. T. Rohde,et al.  Language acquisition in the absence of explicit negative evidence: how important is starting small? , 1999, Cognition.

[17]  Peter M. Vishton,et al.  Rule learning by seven-month-old infants. , 1999, Science.

[18]  Jürgen Schmidhuber,et al.  LSTM recurrent networks learn simple context-free and context-sensitive languages , 2001, IEEE Trans. Neural Networks.

[19]  J. Elman.  Learning and development in neural networks: the importance of starting small , 1993, Cognition.

[20]  Ron Kohavi,et al.  Supervised and Unsupervised Discretization of Continuous Features , 1995, ICML.