Uncertainty About the Rest of the Sentence

A word-by-word human sentence processing complexity metric is presented. This metric formalizes the intuition that comprehenders have more trouble on words contributing larger amounts of information about the syntactic structure of the sentence as a whole. The formalization is in terms of the conditional entropy of grammatical continuations, given the words that have been heard so far. To calculate the predictions of this metric, Wilson and Carroll's (1954) original entropy reduction idea is extended to infinite languages. This is demonstrated with a mildly context-sensitive language that includes relative clauses formed on a variety of grammatical relations across the Accessibility Hierarchy of Keenan and Comrie (1977). Predictions are derived that correlate significantly with repetition accuracy results obtained in a sentence-memory experiment (Keenan & Hawkins, 1987).

[1]  Claude E. Shannon,et al.  Prediction and Entropy of Printed English , 1951 .

[2]  Wilson L. Taylor,et al.  “Cloze Procedure”: A New Tool for Measuring Readability , 1953 .

[3]  Frieda Goldman-Eisler,et al.  Speech Production and the Predictability of Words in Context , 1958 .

[4]  Victor H. Yngve,et al.  A model and an hypothesis for language structure , 1960 .

[5]  G. A. Miller,et al.  Finitary models of language users , 1963 .

[6]  Eugene Galanter,et al.  Handbook of mathematical psychology: I. , 1963 .

[7]  H. Kucera,et al.  Computational analysis of present-day American English , 1967 .

[8]  J. Hayes.  Cognition and the development of language , 1970 .

[9]  Ronald M. Kaplan,et al.  Augmented Transition Networks as Psychological Models of Sentence Comprehension , 1971, IJCAI.

[10]  P. Stanley Peters,et al.  On the generative power of transformational grammars , 1973, Inf. Sci..

[11]  Paul Schachter.  Focus and relativization , 1973 .

[12]  B. Lang.  Deterministic Techniques for Efficient Non-Deterministic Parsers , 1974, ICALP.

[13]  Aravind K. Joshi,et al.  Tree Adjunct Grammars , 1975, J. Comput. Syst. Sci..

[14]  Michael K. Brame.  Conjectures and refutations in syntax and semantics , 1976 .

[15]  Noam Chomsky,et al.  On Wh-Movement , 1977 .

[16]  H E Wanner,et al.  An ATN approach to comprehension , 1978 .

[17]  James Mccloskey.  The Syntax of Relative Clauses , 1979 .

[18]  G. Miller,et al.  Cognitive science. , 1981, Science.

[19]  G. Miller,et al.  Linguistic theory and psychological reality , 1982 .

[20]  Aravind K. Joshi,et al.  Natural language parsing: Tree adjoining grammars: How much context-sensitivity is required to provide reasonable structural descriptions? , 1985 .

[21]  M. Baltin,et al.  The Mental representation of grammatical relations , 1985 .

[22]  Anthony S. Kroch,et al.  The Linguistic Relevance of Tree Adjoining Grammar , 1985 .

[23]  Martin Kay,et al.  Algorithm schemata and data structures in syntactic processing , 1986 .

[24]  Karen Spärck Jones,et al.  Readings in natural language processing , 1986 .

[25]  David J. Weir,et al.  Characterizing mildly context-sensitive grammar formalisms , 1988 .

[26]  Bernard Lang,et al.  Parsing Incomplete Sentences , 1988, COLING.

[27]  Bernard Lang,et al.  The Structure of Shared Forests in Ambiguous Parsing , 1989, ACL.

[28]  David J. Weir,et al.  The convergence of mildly context-sensitive grammar formalisms , 1990 .

[29]  Edward Gibson,et al.  A computational theory of human linguistic processing: memory limitations and processing breakdown , 1991 .

[30]  Mark Johnson,et al.  Variations on incremental interpretation , 1993, Journal of Psycholinguistic Research.

[31]  Richard S. Kayne.  The Antisymmetry of Syntax , 1994 .

[32]  Ivan A. Sag,et al.  Book Reviews: Head-driven Phrase Structure Grammar and German in Head-driven Phrase-structure Grammar , 1996, CL.

[33]  Ann Bies,et al.  The Penn Treebank: Annotating Predicate Argument Structure , 1994, HLT.

[34]  John A. Hawkins,et al.  A Performance Theory of Order and Constituency , 1995 .

[35]  John A. Hawkins.  A Performance Theory of Order and Constituency: Envoi , 1995 .

[36]  Edward P. Stabler,et al.  Derivational Minimalism , 1996, LACL.

[37]  Daniel Jurafsky,et al.  A Probabilistic Model of Lexical and Syntactic Access and Disambiguation , 1996, Cogn. Sci..

[38]  I. Sag.  English relative clause constructions , 1997, Journal of Linguistics.

[39]  R. Borsley.  Remarks and replies , 1997 .

[40]  Jens Michaelis,et al.  Derivational Minimalism Is Mildly Context-Sensitive , 1998, LACL.

[41]  E. Gibson.  Linguistic complexity: locality of syntactic dependencies , 1998, Cognition.

[42]  J. Zwart.  The Minimalist Program , 1998, Journal of Linguistics.

[43]  Judy B. Bernstein,et al.  How children's relatives solve a problem for minimalism , 1998 .

[44]  G. Altmann.  Ambiguity in sentence processing , 1998, Trends in Cognitive Sciences.

[45]  Martin J. Pickering,et al.  The rational of analysis of inquiry: The case of parsing. , 1998 .

[46]  N. Pearlmutter,et al.  Serial versus Parallel Sentence Comprehension , 1998 .

[47]  Zhiyi Chi,et al.  Statistical Properties of Probabilistic Context-Free Grammars , 1999, CL.

[48]  Mark Steedman,et al.  The nite connectivity of linguistic structure , 1999 .

[49]  Glyn Morrill,et al.  Incremental processing and acceptability , 2000, CL.

[50]  Y. Miyashita,et al.  Image, language, brain , 2000 .

[51]  E. Gibson.  The dependency locality theory: A distance-based theory of linguistic complexity. , 2000 .

[52]  V. Bianchi.  The Raising Analysis of Relative Clauses: A Reply to Borsley , 2000, Linguistic Inquiry.

[53]  Daniel Jurafsky,et al.  A Bayesian Model Predicts Human Parse Preference and Reading Times in Sentence Processing , 2001, NIPS.

[54]  Gertjan van Noord.  Robust Parsing of Word Graphs , 2001 .

[55]  Jens Michaelis,et al.  On Formal Properties of Minimalist Grammars , 2001 .

[56]  David C. Plaut,et al.  A connectionist model of sentence comprehension and production , 2002 .

[57]  Joseph Aoun,et al.  Essays on the Representational and Derivational Nature of Grammar: The Diversity of Wh-Constructions , 2003 .

[58]  John Hale,et al.  The Information Conveyed by Words in Sentences , 2003, Journal of psycholinguistic research.

[59]  P. Smolensky,et al.  Optimality in Sentence Processing , 2003 .

[60]  Edward P. Stabler,et al.  Structural similarity within and among languages , 2003, Theor. Comput. Sci..

[61]  Daniel C. Richardson,et al.  Effects of merely local syntactic coherence on sentence processing , 2004 .

[62]  R. Frank.  Restricting grammatical complexity , 2004, Cogn. Sci..

[63]  Edward P. Stabler.  Varieties of crossing dependencies: structure dependence and mild context sensitivity , 2004 .

[64]  John Hale,et al.  Grammar, uncertainty and sentence processing , 2004 .

[65]  Edward P. Stabler,et al.  Strict Deterministic Aspects of Minimalist Grammars , 2005, LACL.

[66]  Richard L. Lewis,et al.  An Activation-Based Model of Sentence Processing as Skilled Memory Retrieval , 2005, Cogn. Sci..

[67]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[68]  Alaa A. Kharbouch,et al.  Three models for the description of language , 1956, IRE Trans. Inf. Theory.

[69]  Daniel Götzmann.  Multiple Context-Free Grammars , 2007 .

[70]  E. Keenan,et al.  Noun Phrase Accessibility and Universal Grammar , 2008 .