On the mental representations originating during the interaction between language and vision

The interaction between vision and language processing is clearly of interest to both cognitive psychologists and psycholinguists. Recent research has begun to create understanding of the interaction between vision and language in terms of the representational issues involved. In this paper, we first review some of the theoretical and methodological issues in the current vision–language interaction debate. Later, we develop a model that attempts to account for effects of affordances and visual context on language-scene interaction as well as the role of sensorimotor simulation. The paper addresses theoretical issues related to the mental representations that arise when visual and linguistic systems interact.

[1]  G. Altmann,et al.  The real-time mediation of visual attention by language and world knowledge: Linking anticipatory (and other) eye movements to linguistic processing , 2007 .

[2]  R. Mishra.  Interaction of language and visual attention: evidence from production and comprehension. , 2009, Progress in brain research.

[3]  D. Purves.  Principles of Cognitive Neuroscience , 2007 .

[4]  Mike Rinck,et al.  Episodic affordances contribute to language comprehension , 2009, Language and Cognition.

[5]  Rachel S. Sussman,et al.  Structural and semantic constraints on the resolution of pronouns and reflexives , 2009, Cognition.

[6]  Julie C. Sedivy,et al.  Achieving incremental semantic interpretation through contextual representation , 1999, Cognition.

[7]  Alfonso Caramazza,et al.  Engaging the motor system with masked orthographic primes: A kinematic analysis , 2008 .

[8]  Julie C. Sedivy,et al.  Eye movements to pictures reveal transient semantic activation during spoken word recognition. , 2006, Journal of experimental psychology. Learning, memory, and cognition.

[9]  C. Madden,et al.  Reading between the lines: The activation of background knowledge during text comprehension , 2009 .

[10]  G. Altmann.  Language-mediated eye movements in the absence of a visual world: the ‘blank screen paradigm’ , 2004, Cognition.

[11]  G. Zelinsky.  A theory of eye movements during target acquisition. , 2008, Psychological review.

[12]  Christine D. Wilson,et al.  Grounding conceptual knowledge in modality-specific systems , 2003, Trends in Cognitive Sciences.

[13]  E. Reed.  The Ecological Approach to Visual Perception , 1989 .

[14]  Anna Papafragou,et al.  Does language guide event perception? Evidence from eye movements , 2008, Cognition.

[15]  J. Henderson,et al.  High-level scene perception. , 1999, Annual review of psychology.

[16]  Yuki Kamide,et al.  Now you see it, now you don't: mediating the mapping between language and the visual world , 2004 .

[17]  P. Perona,et al.  What do we perceive in a glance of a real-world scene? , 2007, Journal of vision.

[18]  Laurence R. Harris,et al.  Vision and Attention , 2001 .

[19]  J. Henderson.  Regarding Scenes , 2007 .

[20]  T. Givón.  4. Human language as an evolutionary product , 2002 .

[21]  Paul D. Allopenna,et al.  Tracking the Time Course of Spoken Word Recognition Using Eye Movements: Evidence for Continuous Mapping Models , 1998 .

[22]  David Crundall,et al.  Confirming statements about pictures of natural scenes , 2005 .

[23]  Jillian H. Fecteau,et al.  Priming of pop-out depends upon the current goals of observers. , 2007, Journal of vision.

[24]  M. Giese,et al.  Nonvisual Motor Training Influences Biological Motion Perception , 2006, Current Biology.

[25]  A. Greenwald,et al.  Sensory feedback mechanisms in performance control: with special reference to the ideo-motor mechanism. , 1970, Psychological review.

[26]  G. Altmann,et al.  Discourse-mediation of the mapping between language and the visual world: Eye movements and mental representation , 2009, Cognition.

[27]  Rolf A. Zwaan,et al.  Seeing, acting, understanding: motor resonance in language comprehension. , 2006, Journal of experimental psychology. General.

[28]  J. Henderson.  Human gaze control during real-world scene perception , 2003, Trends in Cognitive Sciences.

[29]  Fernando Marmolejo-Ramos,et al.  NUEVOS AVANCES EN EL ESTUDIO CIENTÍFICO DE LA COMPRENSIÓN DE TEXTOS , 2007 .

[30]  K. Verfaillie,et al.  Attentional Orienting and Scene Semantics , 2001 .

[31]  Matthew W. Crocker,et al.  The Coordinated Interplay of Scene, Utterance, and World Knowledge: Evidence From Eye Tracking , 2006, Cogn. Sci..

[32]  A. Noë,et al.  A sensorimotor account of vision and visual consciousness. , 2001, The Behavioral and brain sciences.

[33]  Michael J. Spivey,et al.  The Continuity Of Mind , 2008 .

[34]  Michael J. Spivey,et al.  Oculomotor mechanisms activated by imagery and memory: eye movements to absent objects , 2001, Psychological research.

[35]  Yuki Kamide,et al.  Anticipatory Processes in Sentence Processing , 2008, Lang. Linguistics Compass.

[36]  Teenie Matlock,et al.  The integration of figurative language and static depictions: An eye movement study of fictive motion , 2007, Cognition.

[37]  Christoph Scheepers,et al.  Priming ditransitive structures in comprehension , 2007, Cognitive Psychology.

[38]  M. Tanenhaus,et al.  Linguistic Gender and Spoken-Word Recognition in French , 2000 .

[39]  John M. Findlay,et al.  Visual Attention: The Active Vision Perspective , 2001 .

[40]  Michael J. Spivey,et al.  Gradiency and Visual Context in Syntactic Garden-Paths. , 2007, Journal of memory and language.

[41]  Jelena Mirkovic,et al.  Incrementality and Prediction in Human Sentence Processing , 2009, Cogn. Sci..

[42]  Stephan P. Swinnen,et al.  Interaction of sound and sight during action perception: Evidence for shared modality-dependent action representations , 2009, Neuropsychologia.

[43]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[44]  C. Koch,et al.  Computational modelling of visual attention , 2001, Nature Reviews Neuroscience.

[45]  Julie C. Sedivy,et al.  Subject Terms: Linguistics Language Eyes & eyesight Cognition & reasoning , 1995 .

[46]  Michelle R. Greene,et al.  Recognition of natural scenes from global properties: Seeing the forest without representing the trees , 2009, Cognitive Psychology.

[47]  Julie C. Sedivy,et al.  Eye movements as a window into real-time spoken language comprehension in natural contexts , 1995, Journal of psycholinguistic research.

[48]  Roger M. Cooper,et al.  The control of eye fixation by the meaning of spoken language: A new methodology for the real-time investigation of speech perception, memory, and language processing. , 1974 .

[49]  Michael J. Spivey,et al.  The enactment of language: Decades of interactions between linguistic and motor processes , 2009, Language and Cognition.

[50]  K. Rayner.  The 35th Sir Frederick Bartlett Lecture: Eye movements and attention in reading, scene perception, and visual search , 2009, Quarterly journal of experimental psychology.

[51]  Graham L. Pierce,et al.  Eye movements during scene viewing: Evidence for mixed control of fixation durations , 2008, Psychonomic bulletin & review.

[52]  M. Tanenhaus,et al.  Circumscribing Referential Domains during Real-Time Language Comprehension , 2002 .

[53]  K. Rayner.  Eye movements in reading and information processing: 20 years of research. , 1998, Psychological bulletin.

[54]  Julie C. Sedivy,et al.  Eye movements and spoken language comprehension: Effects of visual context on syntactic ambiguity resolution , 2002, Cognitive Psychology.

[55]  Michael P. Kaschak,et al.  Constructing Meaning: The Role of Affordances and Grammatical Constructions in Sentence Comprehension , 2000 .

[56]  Carrick C. Williams,et al.  Eye movements during information processing tasks: Individual differences and cultural effects , 2007, Vision Research.

[57]  L. Barsalou.  Grounded cognition. , 2008, Annual review of psychology.

[58]  U. Castiello.  Mechanisms of selection for the control of hand action , 1999, Trends in Cognitive Sciences.

[59]  Massimo Vescovi,et al.  A kinematic approach to the conceptual representations of this and that , 2009, Cognition.

[60]  G. Altmann,et al.  The time-course of prediction in incremental sentence processing: Evidence from anticipatory eye-movements , 2003 .

[61]  Fernanda Ferreira,et al.  Scene Perception for Psycholinguists. , 2004 .

[62]  Falk Huettig,et al.  Visual-shape competition during language-mediated attention is based on lexical input and not modulated by contextual appropriateness , 2007 .

[63]  Michael P. Kaschak,et al.  Putting words in perspective , 2004, Memory & cognition.

[64]  Ray Jackendoff,et al.  A Parallel Architecture perspective on language processing , 2007, Brain Research.

[65]  A. L. I︠A︡rbus.  Eye Movements and Vision , 1967 .

[66]  Ramesh Kumar Mishra,et al.  Simulating Motion in Figurative Language Comprehension , 2010, The open neuroimaging journal.

[67]  T. Shultz,et al.  Modeling developmental cognitive neuroscience , 2006, Trends in Cognitive Sciences.

[68]  Benoit Lemaire.  Models of High-dimensional Semantic Spaces , 1998 .

[69]  Matthew W. Crocker,et al.  The influence of recent scene events on spoken comprehension: Evidence from eye movements , 2007 .

[70]  Antonio Torralba,et al.  Contextual guidance of eye movements and attention in real-world scenes: the role of global features in object search. , 2006, Psychological review.

[71]  Matthew W. Crocker,et al.  The influence of the immediate visual context on incremental thematic role-assignment: evidence from eye-movements in depicted events , 2005, Cognition.

[72]  L. Barsalou,et al.  Whither structured representation? , 1999, Behavioral and Brain Sciences.

[73]  W. Prinz,et al.  Perceptual resonance: action-induced modulation of perception , 2007, Trends in Cognitive Sciences.

[74]  A. Glenberg,et al.  What memory is for: Creating meaning in the service of action , 1997, Behavioral and Brain Sciences.

[75]  Michael P. Kaschak,et al.  Perception of motion affects language processing , 2005, Cognition.

[76]  N. Mackworth,et al.  The gaze selects informative details within pictures , 1967 .

[77]  M. Tanenhaus,et al.  Actions and affordances in syntactic ambiguity resolution. , 2004, Journal of experimental psychology. Learning, memory, and cognition.

[78]  Marshall R. Mayberry,et al.  Learning to Attend: A Connectionist Model of Situated Language Comprehension , 2009, Cogn. Sci..

[79]  G. Altmann,et al.  Incremental interpretation at verbs: restricting the domain of subsequent reference , 1999, Cognition.

[80]  D. Bub,et al.  Kicking Calculators: Contribution of Embodied Representations to Sentence Comprehension. , 2008 .

[81]  Arthur M. Glenberg,et al.  Indexical understanding of instructions , 1999 .

[82]  Linda Lundström,et al.  The pupils and optical systems of gecko eyes. , 2009, Journal of vision.

[83]  I. Biederman.  Perceiving Real-World Scenes , 1972, Science.

[84]  D. Roy.  Grounding words in perception and action: computational insights , 2005, Trends in Cognitive Sciences.

[85]  Qing Yang,et al.  Effect of Title on Eye-Movement Exploration of Cubist Paintings by Fernand Léger , 2009, Perception.

[86]  I. Johnsrude,et al.  Somatotopic Representation of Action Words in Human Motor and Premotor Cortex , 2004, Neuron.

[87]  G. Vingerhoets,et al.  Conceptual and physical object qualities contribute differently to motor affordances , 2009, Brain and Cognition.

[88]  Nancy Millette,et al.  How People Look at Pictures , 1935 .

[89]  David Crundall,et al.  Confirming Statements about Pictures of Natural Scenes: Evidence of the Processing of Gist from Eye Movements , 2005, Perception.

[90]  G. Underwood.  Cognitive processes in eye guidance , 2005 .

[91]  T. Givón.  1. The visual information-processing system as an evolutionaryprecursor of human language , 2002 .

[92]  Sotaro Shimada,et al.  Deactivation in the sensorimotor area during observation of a human agent performing robotic actions , 2010, Brain and Cognition.

[93]  N. Mackworth,et al.  Cognitive determinants of fixation location during picture viewing. , 1978, Journal of experimental psychology. Human perception and performance.

[94]  Alice H. D. Chan,et al.  Language affects patterns of brain activation associated with perceptual decision , 2008, Proceedings of the National Academy of Sciences.

[95]  M. Tanenhaus,et al.  Introduction to the special issue on language–vision interactions , 2007 .

[96]  Marshall R. Mayberry,et al.  Situated sentence processing: The coordinated interplay account and a neurobehavioral model , 2010, Brain and Language.

[97]  George L. Malcolm,et al.  Eye Movements and Visual Encoding During Scene Perception , 2009, Psychological science.

[98]  Derrick J. Parkhurst,et al.  Scene content selected by active vision. , 2003, Spatial vision.

[99]  J. J. Gibson.  The theory of affordances , 1977 .

[100]  Michael L. Mack,et al.  Viewing task influences eye movement control during active scene perception. , 2009, Journal of vision.