Separating learning and representation

Two of the most promising aspects of connectionist natural language research have been (i) the use of powerful statistical learning techniques to model language learning and (ii) the development of new representational theories. Often the two are treated together; some part of a grammar is induced by a net and the subsequent representations are analysed for the maintenance of structural information. In this chapter, representation and learning are treated separately. A simple recurrent net trained on a bidirectional link grammar showed severe limitations in its ability to handle embedded sequences. Then, after an analysis of the problem, a constructive method was used to develop representations, using the same SRN architecture, that exhibited the potential to correctly recognise embeddings of any length. These findings illustrate the benefits of the study of representation, which can provide a basis for the development of novel learning rules.

[1]  David J. Chalmers,et al.  Syntactic Transformations on Distributed Representations , 1990 .

[2]  Geoffrey E. Hinton,et al.  Learning distributed representations of concepts. , 1989 .

[3]  Jeffrey L. Elman,et al.  Distributed Representations, Simple Recurrent Networks, and Grammatical Structure , 1991, Mach. Learn..

[4]  P. Smolensky.  On the proper treatment of connectionism , 1988, Behavioral and Brain Sciences.

[5]  Douglas S. Blank,et al.  Exploring the Symbolic/Subsymbolic Continuum: A case study of RAAM , 1992 .

[6]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[7]  Noel E. Sharkey,et al.  Opening the black box of connectionist nets: Some lessons from cognitive science , 1994 .

[8]  Axel Cleeremans,et al.  Mechanisms of Implicit Learning: Connectionist Models of Sequence Processing , 1993 .

[9]  J. Fodor,et al.  Connectionism and the problem of systematicity: Why Smolensky's solution doesn't work , 1990, Cognition.

[10]  Terrence J. Sejnowski,et al.  NETtalk: a parallel network that learns to read aloud , 1988 .

[11]  Robert F. Hadley.  Systematicity in Connectionist Language Learning , 1994 .

[12]  Geoffrey E. Hinton,et al.  Parallel Models of Associative Memory , 1989 .

[13]  Jordan B. Pollack,et al.  Recursive Distributed Representations , 1990, Artif. Intell..

[14]  Noel E. Sharkey,et al.  An Internal Report for Connectionists , 1995 .

[15]  Halbert White,et al.  Artificial Neural Networks: Approximation and Learning Theory , 1992 .

[16]  James L. McClelland,et al.  On learning the past-tenses of English verbs: implicit rules or parallel distributed processing , 1986 .

[17]  V. Marchman,et al.  U-shaped learning and frequency effects in a multi-layered perception: Implications for child language acquisition , 1991, Cognition.

[18]  J. Fodor,et al.  Connectionism and cognitive architecture: A critical analysis , 1988, Cognition.

[19]  Lonnie Chrisman,et al.  Learning Recursive Distributed Representations for Holistic Computation , 1991 .

[20]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[21]  Lawrence D. Jackel,et al.  Large Automatic Learning, Rule Extraction, and Generalization , 1987, Complex Syst..

[22]  Antinus Nijholt,et al.  Connectionism and Natural Language Processing , 1992 .

[23]  Lars Niklasson,et al.  Can Connectionist Models Exhibit Non-Classical Structure Sensitivity? , 2019, Proceedings of the Sixteenth Annual Conference of the Cognitive Science Society.

[24]  Lars Niklasson,et al.  Connectionism and the Issues of Compositionality and Systematicity , 1992 .

[25]  James L. McClelland,et al.  An interactive activation model of context effects in letter perception: I. An account of basic findings. , 1981 .

[26]  R. Reilly,et al.  Connectionist approaches to natural language processing , 1994 .

[27]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[28]  Tim van Gelder,et al.  Compositionality: A Connectionist Variation on a Classical Theme , 1990, Cogn. Sci..

[29]  Noel E. Sharkey,et al.  Connectionist Natural Language Processing , 1992 .