An amodal shared resource model of language-mediated visual attention

Language-mediated visual attention describes the interaction of two fundamental components of the human cognitive system, language and vision. Within this paper we present an amodal shared resource model of language-mediated visual attention that offers a description of the information and processes involved in this complex multimodal behavior and a potential explanation for how this ability is acquired. We demonstrate that the model is not only sufficient to account for the experimental effects of Visual World Paradigm studies but also that these effects are emergent properties of the architecture of the model itself, rather than requiring separate information processing channels or modular processing systems. The model provides an explicit description of the connection between the modality-specific input from language and vision and the distribution of eye gaze in language-mediated visual attention. The paper concludes by discussing future applications for the model, specifically its potential for investigating the factors driving observed individual differences in language-mediated eye gaze.

[1]  Christine D. Wilson,et al.  Grounding conceptual knowledge in modality-specific systems , 2003, Trends in Cognitive Sciences.

[2]  R. Mishra,et al.  Mechanisms and Representations of Language-Mediated Visual Attention , 2011, Front. Psychology.

[3]  Chris McNorgan,et al.  An attractor model of lexical conceptual processing: simulating semantic priming , 1999, Cogn. Sci..

[4]  Emily J. Mayberry,et al.  Coherent concepts are computed in the anterior temporal lobes , 2010, Proceedings of the National Academy of Sciences.

[5]  Malte C. Viebahn,et al.  Tracking recognition of spoken words by tracking looks to printed words , 2007, Quarterly journal of experimental psychology.

[6]  Ken McRae,et al.  Category - Specific semantic deficits , 2008 .

[7]  P. Bloom.  How children learn the meanings of words , 2000 .

[8]  Sergio E. Chaigneau,et al.  THE SIMILARITY-IN-TOPOGRAPHY PRINCIPLE: RECONCILING THEORIES OF CONCEPTUAL DEFICITS , 2003, Cognitive neuropsychology.

[9]  Falk Huettig,et al.  Visual-shape competition during language-mediated attention is based on lexical input and not modulated by contextual appropriateness , 2007 .

[10]  James L. McClelland,et al.  Are there mental lexicons? The role of semantics in lexical decision , 2010, Brain Research.

[11]  Michael J. Spivey,et al.  The Continuity Of Mind , 2008 .

[12]  Robert J Hartsuiker,et al.  Looking, language, and memory: bridging research from the visual world and visual search paradigms. , 2011, Acta psychologica.

[13]  Padraic Monaghan,et al.  Modelling the Effects of Formal Literacy Training on Language Mediated Visual Attention , 2013, CogSci.

[14]  James L. McClelland,et al.  A single-system account of semantic and lexical deficits in five semantic dementia patients , 2008, Cognitive neuropsychology.

[15]  Marc F. Joanisse,et al.  Specific phonological impairments in dyslexia revealed by eyetracking , 2006, Cognition.

[16]  A. Meyer,et al.  Using the visual world paradigm to study language processing: a review and critical evaluation. , 2011, Acta psychologica.

[17]  James L. McClelland,et al.  The TRACE model of speech perception , 1986, Cognitive Psychology.

[18]  M. Tanenhaus,et al.  Looking at the rope when looking for the snake: Conceptually mediated eye movements during spoken-word recognition , 2005, Psychonomic bulletin & review.

[19]  Chris McNorgan,et al.  Integrating conceptual knowledge within and across representational modalities , 2011, Cognition.

[20]  Bradford Z. Mahon,et al.  A critical look at the embodied cognition hypothesis and a new proposal for grounding conceptual content , 2008, Journal of Physiology-Paris.

[21]  S. McDonald,et al.  Models of high-dimensional semantic space predict language-mediated eye movements in the visual world. , 2006, Acta psychologica.

[22]  James L. McClelland,et al.  Structure and deterioration of semantic memory: a neuropsychological and computational investigation. , 2004, Psychological review.

[23]  Michael J. Spivey,et al.  On the temporal dynamics of language-mediated vision and vision-mediated language. , 2011, Acta psychologica.

[24]  J. Halberda,et al.  Is this a dax which I see before me? Use of the logical argument disjunctive syllogism supports word-learning in children and adults , 2006, Cognitive Psychology.

[25]  E. Jefferies,et al.  Anterior temporal lobes mediate semantic representation: Mimicking semantic dementia by using rTMS in normal participants , 2007, Proceedings of the National Academy of Sciences.

[26]  Roger M. Cooper,et al.  The control of eye fixation by the meaning of spoken language: A new methodology for the real-time investigation of speech perception, memory, and language processing. , 1974 .

[27]  M. L. Lambon Ralph,et al.  Generalization and Differentiation in Semantic Memory , 2008, Annals of the New York Academy of Sciences.

[28]  Guy Dove.  Beyond perceptual symbols: A call for representational pluralism , 2009, Cognition.

[29]  D. Plaut.  Graded modality-specific specialisation in semantics: A computational account of optic aphasia , 2002, Cognitive neuropsychology.

[30]  J. Ziegler,et al.  Reading acquisition, developmental dyslexia, and skilled reading across languages: a psycholinguistic grain size theory. , 2005, Psychological bulletin.

[31]  P. Monaghan,et al.  Integrating constraints for learning word–referent mappings , 2012, Cognition.

[32]  J. McQueen,et al.  Changing only the probability that spoken words will be distorted changes how they are recognized. , 2012, Journal of the Acoustical Society of America.

[33]  Anuenue Kukona,et al.  Impulse Processing: A Dynamical Systems Model of Incremental Eye Movements in the Visual World Paradigm , 2011, Cogn. Sci..

[34]  Sally Andrews,et al.  From inkmarks to ideas : current issues in lexical processing , 2006 .

[35]  Robert L. Goldstone,et al.  Reuniting perception and conception , 1998, Cognition.

[36]  Christopher T. Kello,et al.  Strategic control in word reading: evidence from speeded responding in the tempo-naming task. , 2000, Journal of experimental psychology. Learning, memory, and cognition.

[37]  Falk Huettig,et al.  The tug of war between phonological, semantic and shape information in language-mediated visual search , 2007 .

[38]  Ellen M. Markman,et al.  Constraints on word meaning in early language acquisition , 1994 .

[39]  M. Tanenhaus,et al.  The time course of spoken word learning and recognition: studies with artificial lexicons. , 2003, Journal of experimental psychology. General.

[40]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[41]  Paul D. Allopenna,et al.  Tracking the Time Course of Spoken Word Recognition Using Eye Movements: Evidence for Continuous Mapping Models , 1998 .

[42]  G. Lupyan.  Linguistically Modulated Perception and Cognition: The Label-Feedback Hypothesis , 2012, Front. Psychology.

[43]  Walter Kintsch,et al.  Symbol systems and perceptual representations , 2008 .

[44]  Mark S. Seidenberg,et al.  2 Progress in understanding word reading : Data fi tting versus theory building , 2006 .

[45]  Mark S. Seidenberg,et al.  Computing the meanings of words in reading: cooperative division of labor between visual and phonological processes. , 2004, Psychological review.

[46]  D. Mirman,et al.  Dynamics of activation of semantically similar concepts during spoken word recognition , 2009, Memory & cognition.

[47]  G. Altmann,et al.  Word meaning and the control of eye fixation: semantic competitor effects and the visual world paradigm , 2005, Cognition.

[48]  Julie C. Sedivy,et al.  Eye movements to pictures reveal transient semantic activation during spoken word recognition. , 2006, Journal of experimental psychology. Learning, memory, and cognition.

[49]  G. Humphreys,et al.  Modelling direct perceptual constraints on action selection: The Naming and Action Model (NAM) , 2002 .

[50]  D. Norris.  Shortlist: a connectionist model of continuous speech recognition , 1994, Cognition.

[51]  W. Marslen-Wilson.  Functional parallelism in spoken word-recognition , 1987, Cognition.

[52]  Julie C. Sedivy,et al.  Subject Terms: Linguistics Language Eyes & eyesight Cognition & reasoning , 1995 .

[53]  Richard S. J. Frackowiak,et al.  Functional anatomy of a common semantic system for words and pictures , 1996, Nature.

[54]  Falk Huettig,et al.  How yellow is your banana? Toddlers' language-mediated visual search in referent-present tasks. , 2013, Developmental psychology.

[55]  R. Mishra,et al.  Language-Mediated Visual Orienting Behavior in Low and High Literates , 2011, Front. Psychology.

[56]  J. Prinz.  Furnishing the Mind: Concepts and Their Perceptual Basis , 2004 .

[57]  Marshall R. Mayberry,et al.  Learning to Attend: A Connectionist Model of Situated Language Comprehension , 2009, Cogn. Sci..

[58]  M. Bar.  A Cortical Mechanism for Triggering Top-Down Facilitation in Visual Object Recognition , 2003, Journal of Cognitive Neuroscience.

[59]  Max Coltheart,et al.  Are There Lexicons? , 2004, The Quarterly journal of experimental psychology. A, Human experimental psychology.