Natural Language Processing with Subsymbolic Neural Networks

Natural language processing appears on the surface to be a strongly symbolic activity. Words are symbols that stand for objects and concepts in the real world, and they are put together into sentences that obey well-speci ed grammar rules. It is no surprise that for several decades natural language processing research has been dominated by the symbolic approach. Linguists have focused on describing language systems based on versions of the Universal Grammar. Arti cial Intelligence researchers have built large programs where linguistic and world knowledge is expressed in symbolic structures, usually in LISP. Relatively little attention has been paid to various cognitive e ects in language processing. Human language users perform di erently from their linguistic competence, that is, from their knowledge of how to communicate correctly using language. Some linguistic structures (such as deep embeddings) are harder to deal with than others. People make mistakes when they speak, but fortunately it is not that hard to understand language that is ungrammatical or cluttered with errors. Linguistic and symbolic arti cial intelligence theories have little to say about where such e ects come from. Yet if one wants to build machines that would communicate naturally with people, it is important to understand and model cognitive e ects in natural language processing.

[1]  David J. Chalmers,et al.  Syntactic Transformations on Distributed Representations , 1990 .

[2]  Cynthia L. Cosic,et al.  Learning to Represent and Understand Locative Prepositional Phrases , 1988 .

[3]  Jordan B. Pollack,et al.  Recursive Distributed Representations , 1990, Artif. Intell..

[4]  Geoffrey E. Hinton.  Learning and Applying Contextual Constraints in Sentence Comprehension , 1991 .

[5]  Mark F. St. John,et al.  The Story Gestalt: A Model of Knowledge-Intensive Processes in Text Comprehension , 1992, Cogn. Sci..

[6]  Douglas S. Blank,et al.  Exploring the Symbolic/Subsymbolic Continuum: A case study of RAAM , 1992 .

[7]  James L. McClelland,et al.  Mechanisms of Sentence Processing: Assigning Roles to Constituents of Sentences , 1986 .

[8]  David S. Touretzky.  Connectionism and Compositional Semantics , 1989 .

[9]  Jeffrey L. Elman,et al.  A PDP Approach to Processing Center-Embedded Sentences , 1992 .

[10]  Karen Kukich,et al.  Where do Phrases Come from: Some Preliminary Experiments in Connectionist Phrase Generation , 1987 .

[11]  James L. McClelland,et al.  Graded State Machines: The Representation of Temporal Contingencies in Simple Recurrent Networks , 2005, Machine Learning.

[12]  Lonnie Chrisman,et al.  Learning Recursive Distributed Representations for Holistic Computation , 1991 .

[13]  Tim van Gelder,et al.  Compositionality: A Connectionist Variation on a Classical Theme , 1990, Cogn. Sci..

[14]  Barry L. Kalman,et al.  Tail-recursive Distributed Representations and Simple Recurrent Networks , 1995 .

[15]  Risto Miikkulainen,et al.  Natural Language Processingwith Modular Neural Networks and Distributed Lexicon , 1991 .

[16]  Donald J. Foss,et al.  Some effects of memory limitation upon sentence comprehension and recall , 1970 .

[17]  J. Elman.  Distributed Representations, Simple Recurrent Networks, And Grammatical Structure , 1991 .

[18]  Michael Gasser,et al.  Acquiring Receptive Morphology: A Connectionist Model , 1994, ACL.

[19]  Paul W. Munro,et al.  A Network for Encoding, Decoding and Translating Locative Prepositions , 1991 .

[20]  Garrison W. Cottrell,et al.  Acquiring the Mapping from Meaning to Sounds , 1994, Connect. Sci..

[21]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[22]  Walter S. Stolz,et al.  A study of the ability to decode grammatically novel sentences , 1967 .

[23]  Charles J. Fillmore,et al.  THE CASE FOR CASE. , 1967 .

[24]  Risto Miikkulainen.  Subsymbolic Case-Role Analysis of Sentences with Embedded Clauses , 1993 .

[25]  George Berg,et al.  A Connectionist Parser with Recursive Sentence Structure and Lexical Disambiguation , 1992, AAAI.

[26]  Tim van Gelder,et al.  Defining ‘Distributed Representation’ , 1992 .

[27]  A. Caramazza,et al.  Dissociation of algorithmic and heuristic processes in language comprehension: Evidence from aphasia , 1976, Brain and Language.

[28]  Risto Miikkulainen,et al.  Subsymbolic natural language processing - an integrated model of scripts, lexicon, and memory , 1993, Neural network modeling and connectionism.

[29]  Robert B. Allen,et al.  Several Studies on Natural Language ·and Back-Propagation , 1987 .

[30]  Lorraine F. R. Karen.  Identification of Topical Entities in Discourse: a Connectionist Approach to Attentional Mechanisms in Language , 1990 .

[31]  Ajay N. Jain,et al.  Parsing Complex Sentences with Structured Connectionist Networks , 1991, Neural Computation.

[32]  M. Huang.  A Developmental Study of Children's Comprehension of Embedded Sentences with and without Semantic Constraints , 1983 .

[33]  Walter Anthony Cook.  Case Grammar Theory , 1979 .

[34]  George A. Miller,et al.  Free Recall of Self-Embedded English Sentences , 1964, Inf. Control..