Connectionist Sentence Processing in Perspective

The emphasis in the connectionist sentence-processing literature on distributed representation and emergence of grammar from such systems can easily obscure the often close relations between connectionist and symbolist systems. This paper argues that the Simple Recurrent Network (SRN) models proposed by Jordan (1989) and Elman (1990) are more directly related to stochastic Part-of-Speech (POS) Taggers than to parsers or grammars as such, while auto-associative memory models of the kind pioneered by Longuet–Higgins, Willshaw, Pollack and others may be useful for grammar induction from a network-based conceptual structure as well as for structure-building. These observations suggest some interesting new directions for specifically connectionist sentence processing research, including more efficient representations for finite state machines, and acquisition devices based on a distinctively connectionist basis for grounded symbolist conceptual structure.

[1]  J. Piaget,et al.  The Origins of Intelligence in Children , 1971 .

[2]  E. Markman,et al.  When it is better to receive than to give: Syntactic and conceptual constraints on vocabulary growth , 1994 .

[3]  J. K. Bock.  Syntactic persistence in language production , 1986, Cognitive Psychology.

[4]  S. Freud.  Beyond the Pleasure Principle , 1925 .

[5]  Risto Miikkulainen,et al.  Subsymbolic Parsing of Embedded Structures , 1995 .

[6]  Aravind K. Joshi,et al.  Tree-adjoining grammars and lexicalized grammars , 1992, Tree Automata and Languages.

[7]  Lyn Frazier,et al.  Language Processing and Language Acquisition , 1990 .

[8]  J. Fodor,et al.  Connectionism and the problem of systematicity: Why Smolensky's solution doesn't work , 1990, Cognition.

[9]  Ivan A. Sag,et al.  Information-based syntax and semantics , 1987 .

[10]  Caroline Heycock,et al.  Language Acquisition: Knowledge Representation and Processing , 1999 .

[11]  Alan W. Black.  Finite State Machines from Feature Grammars , 1989, IWPT.

[12]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[13]  James L. McClelland,et al.  Graded state machines: the representation of temporal contingencies in feedback networks , 1995 .

[14]  Noam Chomsky,et al.  वाक्यविन्यास का सैद्धान्तिक पक्ष = Aspects of the theory of syntax , 1965 .

[15]  J. Grimshaw.  Projection, heads, and optimality , 1997 .

[16]  M. Baltin,et al.  The Mental representation of grammatical relations , 1985 .

[17]  George Berg,et al.  A Connectionist Parser with Recursive Sentence Structure and Lexical Disambiguation , 1992, AAAI.

[18]  Kenneth Ward Church.  A Stochastic Parts Program and Noun Phrase Parser for Unrestricted Text , 1988, ANLP.

[19]  Paul Smolensky.  Grammar-based connectionist approaches to language , 1999 .

[20]  T. Landauer,et al.  A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge. , 1997 .

[21]  Robert F. Port,et al.  MIND IN MOTION: , 2019, Dune.

[22]  John C. Trueswell,et al.  The Convergence of Lexicalist Perspectives in Psycholinguistics and Computational Linguistics , 2000 .

[23]  Ken Hale,et al.  On Argument Structure and the Lexical Expression of Syntactic Relations , 1993 .

[24]  E. Mark Gold,et al.  Language Identification in the Limit , 1967, Inf. Control..

[25]  F. Jelinek,et al.  Continuous speech recognition by statistical methods , 1976, Proceedings of the IEEE.

[26]  L. Gleitman.  The Structural Sources of Verb Meanings , 2020, Sentence First, Arguments Afterward.

[27]  J. Elman.  Representation and structure in connectionist models , 1991 .

[28]  Srinivas Bangalore,et al.  Complexity of lexical descriptions and its relevance to partial parsing , 1997 .

[29]  J. Siskind.  A computational study of cross-situational techniques for learning word-to-meaning mappings , 1996, Cognition.

[30]  Kathryn Bock,et al.  Meaning, sound and syntax in english number agreement , 1993 .

[31]  Robert F. Hadley.  Systematicity revisited : reply to Christiansen and Chater and Niklasson and van Gelder , 1994 .

[32]  Nick Chater,et al.  Toward a connectionist model of recursion in human linguistic performance , 1999 .

[33]  Jeffrey L. Elman,et al.  Language as a dynamical system , 1996 .

[34]  Mitchell P. Marcus,et al.  Maximum entropy models for natural language ambiguity resolution , 1998 .

[35]  Ron Sun,et al.  Computational Architectures Integrating Neural And Symbolic Processes , 1994 .

[36]  Robert F. Hadley.  Systematicity in Connectionist Language Learning , 1994 .

[37]  Richard Montague,et al.  ENGLISH AS A FORMAL LANGUAGE , 1975 .

[38]  H. C. LONGUET-HIGGINS,et al.  Non-Holographic Associative Memory , 1969, Nature.

[39]  A Prince,et al.  Optimality: From Neural Networks to Universal Grammar , 1997, Science.

[40]  Mark S. Seidenberg,et al.  Language Acquisition and Use: Learning and Applying Probabilistic Constraints , 1997, Science.

[41]  J. Fodor.  The Language of Thought , 1980 .

[42]  Mark Steedman,et al.  Ambiguity in context: A reply , 1989 .

[43]  J. Piaget.  The origins of intelligence in children, New York (W W Norton) 1963. , 1963 .

[44]  Gary S. Dell,et al.  Connectionist models of language production: lexical access and grammatical encoding , 1999, Cogn. Sci..

[45]  Jason Eisner.  Efficient Generation in Primitive Optimality Theory , 1997, ACL.

[46]  Michael Collins,et al.  Three Generative, Lexicalised Models for Statistical Parsing , 1997, ACL.

[47]  Mark Steedman,et al.  Dependency and Coordination in the Grammar of Dutch and English , 1985 .

[48]  Morten H. Christiansen,et al.  Generalization and connectionist language learning , 1994 .

[49]  Paul Smolensky,et al.  Tensor Product Variable Binding and the Representation of Symbolic Structures in Connectionist Systems , 1990, Artif. Intell..

[50]  Geoffrey E. Hinton,et al.  Generative models for discovering sparse distributed representations. , 1997, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[51]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[52]  G. Altmann.  Cognitive models of speech processing , 1991 .

[53]  M. Tanenhaus,et al.  Context effects in syntactic ambiguity resolution: discourse and semantic influences in parsing reduced relative clauses. , 1993, Canadian journal of experimental psychology = Revue canadienne de psychologie experimentale.

[54]  Bernard Mérialdo,et al.  Tagging English Text with a Probabilistic Model , 1994, CL.

[55]  J. Fodor,et al.  Connectionism and cognitive architecture: A critical analysis , 1988, Cognition.

[56]  Michael I. Jordan.  Serial Order: A Parallel Distributed Processing Approach , 1997 .

[57]  M. Tanenhaus,et al.  Modeling the Influence of Thematic Fit (and Other Constraints) in On-line Sentence Comprehension , 1998 .

[58]  James L. McClelland,et al.  Learning and Applying Contextual Constraints in Sentence Comprehension , 1990, Artif. Intell..

[59]  Mark Steedman,et al.  Surface structure and interpretation , 1996, Linguistic inquiry.

[60]  Eugene Charniak,et al.  Statistical Parsing with a Context-Free Grammar and Word Statistics , 1997, AAAI/IAAI.

[61]  Susan M. Garnsey,et al.  Semantic Influences On Parsing: Use of Thematic Role Information in Syntactic Ambiguity Resolution , 1994 .

[62]  Stuart M. Shieber,et al.  Evidence against the context-freeness of natural language , 1985 .

[63]  James L. McClelland,et al.  Mechanisms of Sentence Processing: Assigning Roles to Constituents of Sentences , 1986 .

[64]  Geoffrey E. Hinton,et al.  Distributed representations and nested compositional structure , 1994 .

[65]  T. Gelder,et al.  On Being Systematically Connectionist , 1994 .

[66]  Stanley Peters,et al.  Cross-Serial Dependencies in Dutch , 1982 .

[67]  Mike Casey,et al.  The Dynamics of Discrete-Time Computation, with Application to Recurrent Neural Networks and Finite State Machine Extraction , 1996, Neural Computation.

[68]  Mark Steedman,et al.  Interaction with context during human sentence processing , 1988, Cognition.

[69]  Gerry T. M. Altmann,et al.  Ambiguity, parsing strategies, and computational models , 1988 .

[70]  Morten H. Christiansen,et al.  Learning to Segment Speech Using Multiple Cues: A Connectionist Model , 1998 .

[71]  S. Freud.  Project for a Scientific Psychology , 1895 .

[72]  M. H. Kelly,et al.  Domain-general abilities applied to domain-specific tasks: Sensitivity to probabilities in perception, cognition, and language☆ , 1994 .

[73]  David S. Touretzky,et al.  Connectionist models : proceedings of the 1990 summer school , 1991 .

[74]  Eric Brill,et al.  A Simple Rule-Based Part of Speech Tagger , 1992, HLT.

[75]  Maryellen C. MacDonald,et al.  A probabilistic constraints approach to language acquisition and processing , 1999, Cogn. Sci..

[76]  M. Tanenhaus,et al.  Verb-specific constraints in sentence processing: Separating effects of lexical preference from garden-paths. , 1993 .

[77]  Bruce Tesar,et al.  An iterative strategy for language learning , 1998 .

[78]  Geoffrey E. Hinton,et al.  Mean field networks that learn to discriminate temporally distorted strings , 1991 .

[79]  Bob Rehder,et al.  How Well Can Passage Meaning be Derived without Using Word Order? A Comparison of Latent Semantic Analysis and Humans , 1997 .

[80]  Srinivas Bangalore,et al.  The Institute For Research In Cognitive Science Disambiguation of Super Parts of Speech ( or Supertags ) : Almost Parsing by Aravind , 1995 .

[81]  M. H. Kelly,et al.  Using sound to solve syntactic problems: the role of phonology in grammatical category assignments. , 1992, Psychological review.

[82]  John S. Bridle,et al.  Neural Networks or Hidden Markov Models for Automatic Speech Recognition: Is there a Choice? , 1992 .

[83]  Michael K. Tanenhaus,et al.  Dynamical models of sentence processing , 1999, Cogn. Sci..

[84]  Geoffrey E. Hinton.  Mapping Part-Whole Hierarchies into Connectionist Networks , 1990, Artif. Intell..

[85]  Tony Plate,et al.  Holographic Reduced Representations: Convolution Algebra for Compositional Distributed Representations , 1991, IJCAI.

[86]  Mark Steedman.  Structure and Intonation , 1991 .

[87]  Geoffrey E. Hinton.  Preface to the Special Issue on Connectionist Symbol Processing , 1990 .

[88]  Barak A. Pearlmutter.  Gradient calculations for dynamic recurrent neural networks: a survey , 1995, IEEE Trans. Neural Networks.

[89]  Mark Steedman,et al.  On not being led up the garden path : The use of context by the psychological syntax processor , 1985 .

[90]  Ivan A. Sag,et al.  Book Reviews: Head-driven Phrase Structure Grammar and German in Head-driven Phrase-structure Grammar , 1996, CL.

[91]  William A. Woods,et al.  Computational Linguistics Transition Network Grammars for Natural Language Analysis , 2022 .

[92]  Wendy K. Wilkins,et al.  Brains evolution and neurolinguistic preconditions , 1995 .

[93]  Geoffrey E. Hinton,et al.  Parallel Models of Associative Memory , 1989 .

[94]  Jordan B. Pollack,et al.  Recursive Distributed Representations , 1990, Artif. Intell..

[95]  Geoffrey E. Hinton.  Connectionist Symbol Processing , 1991 .