Redundancy and reduction: Speakers manage syntactic information density

A principle of efficient language production based on information theoretic considerations is proposed: Uniform Information Density predicts that language production is affected by a preference to distribute information uniformly across the linguistic signal. This prediction is tested against data from syntactic reduction. A single multilevel logit model analysis of naturally distributed data from a corpus of spontaneous speech is used to assess the effect of information density on complementizer that-mentioning, while simultaneously evaluating the predictions of several influential alternative accounts: availability, ambiguity avoidance, and dependency processing accounts. Information density emerges as an important predictor of speakers' preferences during production. As information is defined in terms of probabilities, it follows that production is probability-sensitive, in that speakers' preferences are affected by the contextual probability of syntactic structures. The merits of a corpus-based approach to the study of language production are discussed as well.

[1]  A. Agresti An introduction to categorical data analysis , 1997 .

[2]  P. Resnik Selectional constraints: an information-theoretic model and its computational realization , 1996, Cognition.

[3]  H. H. Clark,et al.  Repeating Words in Spontaneous Speech , 1998, Cognitive Psychology.

[4]  Susan M. Garnsey,et al.  Knowledge of Grammar, Knowledge of Usage: Syntactic Probabilities Affect Pronunciation Variation , 2004 .

[5]  F. Ferreira Choice of Passive Voice is Affected by Verb Type and Animacy , 1994 .

[6]  Susan M. Garnsey,et al.  The Contributions of Verb Bias and Plausibility to the Comprehension of Temporarily Ambiguous Sentences , 1997 .

[7]  Zenzi M. Griffin,et al.  A reversed word length effect in coordinating the preparation and articulation of words in speaking , 2004 .

[8]  G. Dell,et al.  Effect of Ambiguity and Lexical Availability on Syntactic and Lexical Production , 2000, Cognitive Psychology.

[9]  John A. Hawkins,et al.  Why are categories adjacent , 2001 .

[10]  K. Bock An Effect of the Accessibility of Word Forms on Sentence Structures , 1987 .

[11]  Barbara A. Fox,et al.  Relative Clauses in English conversation Relativizers , frequency , and the notion of construction * , 2005 .

[12]  John R. Anderson The Adaptive Character of Thought , 1990 .

[13]  Rena Torres Cacoullos,et al.  On the persistence of grammar in discourse formulas: a variationist study of that , 2009 .

[14]  Michael A Babyak,et al.  What You See May Not Be What You Get: A Brief, Nontechnical Introduction to Overfitting in Regression-Type Models , 2004, Psychosomatic medicine.

[15]  T. Jaeger Phonological Optimization and Syntactic Variation: The Case of Optional That , 2006 .

[16]  Barbara John A Thomas Lohse,et al.  Domain Minimization in English Verb-Particle Constructions , 2004 .

[17]  Nathaniel J. Smith,et al.  Optimal Processing Times in Reading: A Formal Model and Empirical Investigation , 2008 .

[18]  M. Pickering,et al.  Structural priming: a critical review. , 2008, Psychological bulletin.

[19]  T. Givon Functionalism and Grammar , 1995 .

[20]  Eugene Charniak,et al.  Entropy Rate Constancy in Text , 2002, ACL.

[21]  Mercè Prat-Sala,et al.  Discourse constraints on syntactic processing in language production , 2000 .

[22]  R. Langacker Foundations of cognitive grammar , 1983 .

[23]  T. Jaeger,et al.  Speaking Rationally: Uniform Information Density as an Optimal Strategy for Language Production , 2008 .

[24]  Eric Wanner,et al.  Language acquisition: the state of the art , 1982 .

[25]  Z. Feng,et al.  A comparison of statistical methods for clustered data analysis with Gaussian error. , 1996, Statistics in medicine.

[26]  Thomas Wasow,et al.  Remarks on grammatical weight , 1997, Language Variation and Change.

[27]  John Hale,et al.  The Information Conveyed by Words in Sentences , 2003, Journal of psycholinguistic research.

[28]  Randolph Quirk Relative clauses in educated spoken English 1 , 1957 .

[29]  H. H. Clark,et al.  Using uh and um in spontaneous speaking , 2002, Cognition.

[30]  W. Levelt Speaking: From Intention to Articulation , 1990 .

[31]  Mirjam Ernestus,et al.  Articulatory Planning Is Continuous and Sensitive to Informational Redundancy , 2005, Phonetica.

[32]  Michael J Cortese,et al.  Handbook of Psycholinguistics , 2011 .

[33]  Gary S. Dell,et al.  Connectionist models of language production: lexical access and grammatical encoding , 1999, Cogn. Sci..

[34]  Gertraud Fenk-Oczlon Familiarity, information flow, and linguistic form , 2001 .

[35]  Dan Jurafsky,et al.  Effects of disfluencies, predictability, and utterance position on word form variation in English conversation. , 2003, The Journal of the Acoustical Society of America.

[36]  Jean-Christophe Verstraete,et al.  Review of Joan Bybee & Paul J. Hopper (2001) Frequency and the emergence of linguistic structure (Amsterdam: Benjamins) , 2005 .

[37]  D. Bates,et al.  Nonlinear mixed effects models for repeated measures data. , 1990, Biometrics.

[38]  J. K. Bock Syntactic persistence in language production , 1986, Cognitive Psychology.

[39]  W. Kintsch,et al.  Language and comprehension , 1982 .

[40]  Dmitrii Manin,et al.  Experiments on predictability of word in context and information rate in natural language , 2006, ArXiv.

[41]  Tania Avgustinova,et al.  Word order and clitics in Bulgarian , 1997 .

[42]  J. Tebbs,et al.  An Introduction to Categorical Data Analysis , 2008 .

[43]  N. Chater,et al.  Ten years of the rational analysis of cognition , 1999, Trends in Cognitive Sciences.

[44]  T. Jaeger,et al.  Implicit Learning and Syntactic Persistence: Surprisal and Cumulativity , 2007 .

[45]  Elizabeth K. Johnson,et al.  Statistical learning of tone sequences by human infants and adults , 1999, Cognition.

[46]  G. Altmann,et al.  The time-course of prediction in incremental sentence processing: Evidence from anticipatory eye-movements , 2003 .

[47]  David Temperley,et al.  Ambiguity Avoidance in English Relative Clauses , 2003 .

[48]  Andrew Zisserman,et al.  Advances in Neural Information Processing Systems (NIPS) , 2007 .

[49]  Joan L. Bybee,et al.  Word frequency and context of use in the lexical diffusion of phonetically conditioned sound change , 2002, Language Variation and Change.

[50]  Mary Ann Walter,et al.  Repetition Avoidance in Human Language , 2007 .

[51]  J. Hawkins Efficiency and complexity in grammars , 2004 .

[52]  Emmon W. Bach,et al.  Universals in Linguistic Theory , 1970 .

[53]  R. Schiffer Psychobiology of Language , 1986 .

[54]  Christopher T. Kello,et al.  Verb-specific constraints in sentence processing: separating effects of lexical preference from garden-paths. , 1993, Journal of experimental psychology. Learning, memory, and cognition.

[55]  Jan P. H. van Santen,et al.  Duration and spectral balance of intervocalic consonants: A case for efficient communication , 2005, Speech Commun..

[56]  G. Dell,et al.  Becoming syntactic. , 2006, Psychological review.

[57]  H. H. Clark,et al.  Hearers and speech acts , 1982 .

[58]  Sarah Brown-Schmidt,et al.  Little houses and casas pequeñas: Message formulation and syntactic form in unscripted speech with speakers of English and Spanish , 2008, Cognition.

[59]  R. Levy Expectation-based syntactic comprehension , 2008, Cognition.

[60]  Sasha Calhoun,et al.  Information structure and the prosodic structure of English : a probabilistic relationship , 2007 .

[61]  John Hale,et al.  A Probabilistic Earley Parser as a Psycholinguistic Model , 2001, NAACL.

[62]  M. Aylett,et al.  Language redundancy predicts syllabic duration and the spectral characteristics of vocalic syllable nuclei. , 2006, The Journal of the Acoustical Society of America.

[63]  Frank E. Harrell,et al.  Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis , 2001 .

[64]  Zenzi M. Griffin,et al.  The persistence of structural priming: transient activation or implicit learning? , 2000, Journal of experimental psychology. General.

[65]  H. D. Adamson,et al.  Social and Processing Constraints on Relative Clauses , 1992 .

[66]  J. Elman,et al.  Why is that? Structural prediction and ambiguity resolution in a very large corpus of English sentences , 2006, Cognition.

[67]  M. Harding,et al.  Using a Laplace Approximation to Estimate the Random Coefficients Logit Model By Nonlinear Least Squares , 2006 .

[68]  John A. Hawkins,et al.  A Performance Theory of Order and Constituency , 1995 .

[69]  W. Levelt,et al.  Word frequency effects in speech production: Retrieval of syntactic information and of phonological form , 1994 .

[70]  Douglas Biber,et al.  Style and Sociolinguistic Variation: Register variation and social dialect variation: the Register Axiom , 2002 .

[71]  Victor S Ferreira,et al.  Proactive interference effects on sentence production , 2002, Psychonomic bulletin & review.

[72]  P. O'Seaghdha,et al.  Phrasal Ordering Constraints in Sentence Production: Phrase Length and Verb Disposition in Heavy-NP Shift , 1998 .

[73]  Morten H. Christiansen,et al.  Experience and sentence processing: Statistical learning and relative clause comprehension , 2009, Cognitive Psychology.

[74]  Sali A. Tagliamonte,et al.  No taming the vernacular! Insights from the relatives in northern Britain , 2005, Language Variation and Change.

[75]  T. Jaeger,et al.  Categorical Data Analysis: Away from ANOVAs (transformation or not) and towards Logit Mixed Models. , 2008, Journal of memory and language.

[76]  R. Jacobs,et al.  Perception of speech reflects optimal use of probabilistic speech cues , 2008, Cognition.

[77]  Jason M. Brenier,et al.  Predictability Effects on Durations of Content and Function Words in Conversational English , 2009 .

[78]  Elizabeth Closs Traugott,et al.  Subjectivity and subjectivisation: Subjectification in grammaticalisation , 1995 .

[79]  T. Florian Jaeger,et al.  Optional that indicates production difficulty: evidence from disfluencies , 2005, DiSS.

[80]  Louis C. W. Pols,et al.  How efficient is speech , 2003 .

[81]  Edward Gibson,et al.  The Communicative Lexicon Hypothesis , 2009 .

[82]  S. Thompson,et al.  A quantitative perspective on the gramaticization of epistemic parentheticals in English , 1991 .

[83]  Gunther Kaltenböck,et al.  ‘… That is the question': complementizer omission in extraposed that-clauses , 2006, English Language and Linguistics.

[84]  G. Schwarz Estimating the Dimension of a Model , 1978 .

[85]  Andrew Gelman,et al.  Data Analysis Using Regression and Multilevel/Hierarchical Models , 2006 .

[86]  David I. Beaver,et al.  Lexical Variation in Relativizer Frequency , 2009 .

[87]  J. Trueswell THE ROLE OF LEXICAL FREQUENCY IN SYNTACTIC AMBIGUITY RESOLUTION , 1996 .

[88]  S. Brennan,et al.  Prosodic disambiguation of syntactic structure: For the speaker or for the addressee? , 2005, Cognitive Psychology.

[89]  Johan Elsness That or zero? A look at the choice of object clause connective in a corpus of American English 1 , 1984 .

[90]  T. Florian Jaeger,et al.  Constraints on Optional that : A Strong Word Form OCP Effect , 2005 .

[91]  Mark Steedman,et al.  A Framework for Annotating Information Structure in Discourse , 2005, FCA@ACL.

[92]  John Fry,et al.  Ellipsis and wa-marking in Japanese Conversation , 2001 .

[93]  Daniel Dor Toward a Semantic Account of that-Deletion in English , 2005 .

[94]  Yuen Ren Chao,et al.  Human Behavior and the Principle of Least Effort: An Introduction to Human Ecology , 1950 .

[95]  G. Dell,et al.  Adapting production to comprehension: The explicit mention of instruments , 1987, Cognitive Psychology.

[96]  Eugene Charniak,et al.  Variation of Entropy and Parse Trees of Sentences as a Function of the Sentence Number , 2003, EMNLP.

[97]  Thomas Wasow,et al.  Post-verbal constituent ordering in English , 2003 .

[98]  R. Carter,et al.  Cambridge Grammar of English , 2006 .

[99]  C. Clifton,et al.  Syntactic prediction in language comprehension: evidence from either...or. , 2006, Journal of experimental psychology. Learning, memory, and cognition.

[100]  Jean Christophe Verstraeh Frequency and the emergence of linguistic structure , 2005 .

[101]  Susan Wright,et al.  Subjectivity and subjectivisation : linguistic perspectives , 1995 .

[102]  Carlos Gómez Gallo,et al.  Incremental Syntactic Planning across Clauses , 2008 .

[103]  John J. Godfrey,et al.  SWITCHBOARD: telephone speech corpus for research and development , 1992, [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[104]  George Kingsley Zipf,et al.  Relative Frequency as a Determinant of Phonetic Change , 1930 .

[105]  Charles Carpenter Fries American English Grammar , 1940 .

[106]  Jennifer E. Arnold RUNNING HEAD : AVOIDING ATTACHMENT AMBIGUITIES Avoiding Attachment Ambiguities : the Role of Constituent Ordering , 2004 .

[107]  S. Brennan,et al.  THE FEELING OF ANOTHER'S KNOWING : PROSODY AND FILLED PAUSES AS CUES TO LISTENERS ABOUT THE METACOGNITIVE STATES OF SPEAKERS , 1995 .

[108]  S. Thompson,et al.  The discourse conditions for the use of the complementizer that in conversational English , 1991 .

[109]  Thomas Wasow,et al.  Processing as a Source of Accessibility Effects on Variation , 2005 .

[110]  F. Ferreira,et al.  Conceptual accessibility and sentence production in a free word order language (Odawa) , 2005, Cognition.

[111]  Hanjung Lee,et al.  Parallel Optimization in Case Systems: Evidence from Case Ellipsis in Korean* , 2006 .

[112]  P. Eckert,et al.  Style and Sociolinguistic Variation. , 2002 .

[113]  Andreas Stolcke,et al.  Word predictability after hesitations: a corpus-based study , 1996, Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96.

[114]  George Kingsley Zipf,et al.  Human Behaviour and the Principle of Least Effort: an Introduction to Human Ecology , 2012 .

[115]  R. Shillcock,et al.  Eye Movements Reveal the On-Line Computation of Lexical Probabilities During Reading , 2003, Psychological science.

[116]  Jennifer E. Arnold,et al.  If you say thee uh you are describing something hard: the on-line attribution of disfluency during reference comprehension. , 2007, Journal of experimental psychology. Learning, memory, and cognition.

[117]  I. Krämer,et al.  Cognitive foundations of interpretation , 2007 .

[118]  J. Concato,et al.  A simulation study of the number of events per variable in logistic regression analysis. , 1996, Journal of clinical epidemiology.

[119]  P. Boersma,et al.  Empirical Tests of the Gradual Learning Algorithm , 2001, Linguistic Inquiry.

[120]  P. Gordon,et al.  Memory interference during language processing. , 2001, Journal of experimental psychology. Learning, memory, and cognition.

[121]  Florien J. van Beinum,et al.  Efficiency as an organizing principle of natural speech , 1998, ICSLP.

[122]  Thomas Hofmann,et al.  Speakers optimize information density through syntactic reduction , 2007 .

[123]  T. Jaeger,et al.  BULGARIAN WORD ORDER AND THE ROLE OF THE DIRECT OBJECT CLITIC IN LFG , 2002 .

[124]  Victor S. Ferreira,et al.  The persistence of optional complementizer production: Why saying “that” is not saying “that” at all , 2003 .

[125]  T. Givón,et al.  On Understanding Grammar , 1979 .

[126]  E. Steyerberg,et al.  [Regression modeling strategies]. , 2011, Revista espanola de cardiologia.

[127]  Maryellen C. MacDonald,et al.  Probabilistic constraints and syntactic ambiguity resolution , 1994 .

[128]  Elisabeth Norcliffe,et al.  Head-marking in usage and grammar: A study of variation and change in Yucatec Maya , 2009 .

[129]  M. Pickering,et al.  Do Speakers Avoid Ambiguities During Dialogue? , 2005, Psychological science.

[130]  Noam Chomsky Three Factors in Language Design , 2005, Linguistic Inquiry.

[131]  Michiko Yaguchi The function of the non-deictic that in English☆ , 2001 .

[132]  R. Harald Baayen,et al.  Predicting the dative alternation , 2007 .

[133]  G. Dell,et al.  Persistent structural priming from language comprehension to language production , 2007, Cognition.

[134]  K. Bock Regulating mental energy: Performance units in language production , 1992 .

[135]  S. Piantadosi,et al.  Refer efficiently : Use less informative expressions for more predictable meanings , 2009 .

[136]  J. K. Bock,et al.  Conceptual accessibility and syntactic structure in sentence formulation , 1985, Cognition.

[137]  Douglas Roland,et al.  Frequency of Basic English Grammatical Structures: A Corpus Analysis. , 2007, Journal of memory and language.

[138]  A. Rebrova,et al.  The Cambridge grammar of English in the ELT classroom , 2012 .

[139]  J. Bresnan,et al.  Gradient Grammar: An Effect of Animacy on the Syntax of give in New Zealand and American English , 2008 .

[140]  V. Ferreira Ambiguity, Accessibility, and a Division of Labor for Communicative Success. , 2008, Learning and motivation.

[141]  Jeanette K. Gundel,et al.  Cognitive Status and the form of Referring Expressions in Discourse , 1993, The Oxford Handbook of Reference.

[142]  Mandeep K. Dhami,et al.  The role of representative design in an ecological approach to cognition. , 2004, Psychological bulletin.

[143]  H. H. Clark,et al.  Audience Design in Meaning and Reference , 1982 .

[144]  Claude E. Shannon,et al.  A Mathematical Theory of Communications , 1948 .

[145]  M. Landau Redundancy, Rationality, and the Problem of Duplication and Overlap , 1969 .

[146]  Mirjam Ernestus,et al.  Lexical frequency and acoustic reduction in spoken Dutch. , 2005, The Journal of the Acoustical Society of America.

[147]  B. Love,et al.  Putting the psychology back into psychological models: Mechanistic versus rational approaches , 2008, Memory & cognition.

[148]  D. Bolinger,et al.  That's that , 1972 .

[149]  Maryellen C. MacDonald,et al.  The use of "that" in the Production and Comprehension of Object Relative Clauses , 2003 .

[150]  Frank Keller,et al.  The Entropy Rate Principle as a Predictor of Processing Effort: An Evaluation against Eye-tracking Data , 2004, EMNLP.

[151]  Alice Turk,et al.  The Smooth Signal Redundancy Hypothesis: A Functional Explanation for Relationships between Redundancy, Prosodic Prominence, and Duration in Spontaneous Speech , 2004, Language and speech.

[152]  Terence H. Wilbur,et al.  Schuchardt, the neogrammarians, and the transformational theory of phonological change : four essays , 1972 .

[153]  Jean E. Fox Tree,et al.  Pronouncing “the” as “thee” to signal problems in speaking , 1997, Cognition.

[154]  Willem J. M. Levelt,et al.  Lexical search and order of mention in sentence production , 1981 .

[155]  Daniel Jurafsky,et al.  A Probabilistic Model of Lexical and Syntactic Access and Disambiguation , 1996, Cogn. Sci..

[156]  R. Harald Baayen,et al.  Analyzing linguistic data: a practical introduction to statistics using R, 1st Edition , 2008 .

[157]  Willem J. M. Levelt,et al.  Crossing the Boundaries in Linguistics , 1981 .

[158]  Joseph Picone,et al.  Resegmentation of SWITCHBOARD , 1998, ICSLP.

[159]  N. Breslow,et al.  Approximate inference in generalized linear mixed models , 1993 .

[160]  Richard L. Lewis Interference in short-term memory: The magical number two (or three) in sentence processing , 1996, Journal of psycholinguistic research.

[161]  V. Ferreira Is It Better to Give Than to Donate? Syntactic Flexibility in Language Production , 1996 .

[162]  Sali A. Tagliamonte,et al.  No momentary fancy! The zero ‘complementizer’ in English dialects , 2005, English Language and Linguistics.

[163]  R. Baayen,et al.  Morphological influences on the recognition of monosyllabic monomorphemic words , 2006 .

[164]  A. Lahiri,et al.  Prosodic Units in Speech Production , 1997 .