Stored object knowledge and the production of referring expressions: the case of color typicality

When speakers describe objects with atypical properties, do they include these properties in their referring expressions, even when that is not strictly required for unique referent identification? Based on previous work, we predict that speakers mention the color of a target object more often when the object is atypically colored, compared to when it is typical. Taking literature from object recognition and visual attention into account, we further hypothesize that this behavior is proportional to the degree to which a color is atypical, and whether color is a highly diagnostic feature in the referred-to object’s identity. We investigate these expectations in two language production experiments, in which participants referred to target objects in visual contexts. In Experiment 1, we find a strong effect of color typicality: less typical colors for target objects predict higher proportions of referring expressions that include color. In Experiment 2 we manipulated objects with more complex shapes, for which color is less diagnostic, and we find that the color typicality effect is moderated by color diagnosticity: it is strongest for high-color-diagnostic objects (i.e., objects with a simple shape). These results suggest that the production of atypical color attributes results from a contrast with stored knowledge, an effect which is stronger when color is more central to object identification. Our findings offer evidence for models of reference production that incorporate general object knowledge, in order to be able to capture these effects of typicality on determining the content of referring expressions.

[1]  G. A. Miller THE PSYCHOLOGICAL REVIEW THE MAGICAL NUMBER SEVEN, PLUS OR MINUS TWO: SOME LIMITS ON OUR CAPACITY FOR PROCESSING INFORMATION 1 , 1956 .

[2]  E. Rosch,et al.  Family resemblances: Studies in the internal structure of categories , 1975, Cognitive Psychology.

[3]  E. Rosch Cognitive Representations of Semantic Categories. , 1975 .

[4]  David D. McDonald,et al.  Salience: The Key to the Selection Problem in Natural Language Generation , 1982, ACL.

[5]  I. Biederman Recognition-by-components: a theory of human image understanding. , 1987, Psychological review.

[6]  Glyn W. Humphreys,et al.  Cascade processes in picture identification , 1988 .

[7]  T. Pechmann Incremental speech production and referential overspecification , 1989 .

[8]  G W Humphreys,et al.  The Effects of Surface Detail on Object Categorization and Naming , 1989, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[9]  Robert Dale,et al.  Computational Interpretations of the Gricean Maxims in the Generation of Referring Expressions , 1995, Cogn. Sci..

[10]  B. Keysar,et al.  When do speakers take into account common ground? , 1996, Cognition.

[11]  H. H. Clark,et al.  Conceptual pacts and lexical choice in conversation. , 1996, Journal of experimental psychology. Learning, memory, and cognition.

[12]  M. Tarr,et al.  Becoming a “Greeble” Expert: Exploring Mechanisms for Face Recognition , 1997, Vision Research.

[13]  Daniela Mapelli,et al.  The role of color in object recognition: Evidence from visual agnosia , 1997 .

[14]  D. Barr,et al.  The Egocentric Basis of Language Use , 1998 .

[15]  J. Tanaka,et al.  Color diagnosticity in object recognition , 1999, Perception & psychophysics.

[16]  D. Weiskopf,et al.  The role of color in high-level vision , 2001, Trends in Cognitive Sciences.

[17]  Julie C. Sedivy,et al.  Pragmatic Versus Form-Based Accounts of Referential Contrast: Evidence for Effects of Informativity Expectations , 2003, Journal of psycholinguistic research.

[18]  Daniel Kersten,et al.  Is Color an Intrinsic Property of Object Representation? , 2003, Perception.

[19]  Frédéric Landragin,et al.  Saillance physique et saillance cognitive , 2004 .

[20]  M. Pickering,et al.  Toward a mechanistic psychology of dialogue , 2004, Behavioral and Brain Sciences.

[21]  William S. Horton,et al.  Conversational Common Ground and Memory Processes in Language Production , 2005 .

[22]  Mark S. Seidenberg,et al.  Semantic feature production norms for a large set of living and nonliving things , 2005, Behavior research methods.

[23]  Mark W. Becker,et al.  Object-intrinsic oddities draw early saccades. , 2007, Journal of experimental psychology. Human perception and performance.

[24]  T. Jaeger,et al.  Categorical Data Analysis: Away from ANOVAs (transformation or not) and towards Logit Mixed Models. , 2008, Journal of memory and language.

[25]  Jennifer E. Arnold,et al.  Reference production: Production-internal and addressee-oriented processes , 2008 .

[26]  Robert Dale,et al.  Referring Expression Generation through Attribute-Based Heuristics , 2009, ENLG.

[27]  David J. Therriault,et al.  The role of color diagnosticity in object recognition and representation , 2009, Cognitive Processing.

[28]  George L. Malcolm,et al.  The effects of target template specificity on visual search in real-world scenes: evidence from eye movements. , 2009, Journal of vision.

[29]  George L. Malcolm,et al.  Combining top-down processes to guide eye movements during real-world scene search. , 2010, Journal of vision.

[30]  Martin J Pickering,et al.  The use of visual context during the production of referring expressions , 2010, Quarterly journal of experimental psychology.

[31]  K. Petersson,et al.  The role of color information on object recognition: a review and meta-analysis. , 2011, Acta psychologica.

[32]  G. Altmann,et al.  Looking at anything that is green when hearing “frog”: How object surface colour and stored object colour knowledge influence language-mediated overt attention , 2011, Quarterly journal of experimental psychology.

[33]  F. Ferreira,et al.  Over-specified referring expressions impair comprehension: An ERP study , 2011, Brain and Cognition.

[34]  Emiel Krahmer,et al.  Factors causing overspecification in definite descriptions , 2011 .

[35]  Karl Magnus Petersson,et al.  The Influence of Color Information on the Recognition of Color Diagnostic and Noncolor Diagnostic Objects , 2010, The Journal of general psychology.

[36]  Michael C. Frank,et al.  Predicting Pragmatic Reasoning in Language Games , 2012, Science.

[37]  Emiel Krahmer,et al.  Toward a Computational Psycholinguistics of Reference Production , 2012, Top. Cogn. Sci..

[38]  Moreno I. Coco,et al.  Scan Patterns Predict Sentence Production in the Cross-Modal Processing of Visual Scenes , 2012, Cogn. Sci..

[39]  Ielka van der Sluis,et al.  Generation of Referring Expressions: Assessing the Incremental Algorithm , 2012, Cogn. Sci..

[40]  Roger P. G. van Gompel,et al.  Producing Pronouns and Definite Noun Phrases: Do Speakers Use the Addressee's Discourse Model? , 2012, Cogn. Sci..

[41]  Emiel Krahmer,et al.  Computational Generation of Referring Expressions: A Survey , 2012, CL.

[42]  Emiel Krahmer,et al.  The Impact of Colour Difference and Colour Codability on Reference Production , 2012, CogSci.

[43]  Margaret Mitchell Attributes in Visual Object Reference , 2013 .

[44]  Margaret Mitchell,et al.  Generating reference to visible objects , 2013 .

[45]  M. Elsner,et al.  Where's Wally: the influence of visual salience on referring expression generation , 2013, Front. Psychol..

[46]  Martijn Goudbeek,et al.  The Effect of Scene Variation on the Redundant Use of Color in Definite Reference , 2013, Cogn. Sci..

[47]  D. Barr,et al.  Random effects structure for confirmatory hypothesis testing: Keep it maximal. , 2013, Journal of memory and language.

[48]  Kees van Deemter,et al.  Typicality and Object Reference , 2013, CogSci.

[49]  Anna K. Kuhlen,et al.  Language in Dialogue: When Confederates Might Be Hazardous to Your Data , 2022 .

[50]  Aykut Erdem,et al.  Visual saliency estimation by nonlinearly integrating features using region covariances. , 2013, Journal of vision.

[51]  John K. Tsotsos,et al.  50 Years of object recognition: Directions forward , 2013, Comput. Vis. Image Underst..

[52]  Emiel Krahmer,et al.  How perceived distractor distance influences reference production: Effects of perceptual grouping in 2D and 3D scenes , 2014, CogSci.

[53]  Robert Dale,et al.  Referring in dialogue: alignment or construction? , 2014 .

[54]  Timothy M. Gann,et al.  Speaking from experience: audience design as expert performance , 2014 .