Describing visual scenes: Towards a neurolinguistics based on construction grammar

The present paper is part of a larger effort to locate the production and perception of language within the broader context of brain mechanisms for action and perception more generally. Here we model function in terms of the competition and cooperation of schemas. We use the task of describing visual scenes to explore the suitability of Construction Grammar as an appropriate framework for a schema-based linguistics. We recall the early VISIONS model of schema-based computer analysis of static visual scenes and then introduce SemRep as a graphical representation of dynamic visual scenes designed to support the generation of varied descriptions of episodes. We report preliminary results on implementing the production of sentences using Template Construction Grammar (TCG), a new form of construction grammar distinguished by its use of SemRep to express semantics. We summarize data on neural correlates relevant to future work on TCG within the context of neurolinguistics, and show how the relation between SemRep and TCG can serve as the basis for modeling language comprehension.

[1]  C. Fillmore,et al.  Regularity and Idiomaticity in Grammatical Constructions: The Case of Let Alone , 1988 .

[2]  A. Braun,et al.  Activation of Broca’s area during the production of spoken and signed language: a combined cytoarchitectonic mapping and PET analysis , 2003, Neuropsychologia.

[3]  R. Ilmoniemi,et al.  Functional links between motor and language systems , 2005, The European journal of neuroscience.

[4]  M. Goodale,et al.  Separate visual pathways for perception and action , 1992, Trends in Neurosciences.

[5]  Kevin P. Hinshaw,et al.  Functional Roles of Broca's Area and SMG: Evidence from Cortical Stimulation Mapping in a Deaf Signer , 1999, NeuroImage.

[6]  R. Adolphs,et al.  NEURAL CORRELATES OF CONCEPTUAL KNOWLEDGE FOR ACTIONS , 2003, Cognitive neuropsychology.

[7]  K. Amunts,et al.  Broca's region , 2006 .

[8]  P. Goldman-Rakic,et al.  Dissociation of object and spatial processing domains in primate prefrontal cortex. , 1993, Science.

[9]  Richard D. Hichwa,et al.  A neural basis for lexical retrieval , 1996, Nature.

[10]  Michael A. Arbib,et al.  A formal model of computation for sensory-based robotics , 1989, IEEE Trans. Robotics Autom..

[11]  H. Coslett,et al.  Simultanagnosia. To see but not two see. , 1991, Brain : a journal of neurology.

[12]  R. Schmidt.  A schema theory of discrete motor skill learning. , 1975 .

[13]  M. Ullman.  Contributions of memory circuits to language: the declarative/procedural model , 2004, Cognition.

[14]  L. Squire,et al.  The human perirhinal cortex and recognition memory , 1998, Hippocampus.

[15]  François Michel,et al.  Seeing without the Occipito-Parietal Cortex: Simultagnosia as a Shrinkage of the Attentional Visual Field , 2004, Behavioural neurology.

[16]  H. Grice.  Utterer's meaning and intentions , 1969 .

[17]  Michael A. Arbib,et al.  Extending the mirror neuron system model, I , 2007, Biological Cybernetics.

[18]  M. Arbib.  Action to language via the mirror neuron system , 2006 .

[19]  Jean Piaget,et al.  Biologie et connaissance : essai sur les relations entre les régulations organiques et les processus cognitifs , 1992 .

[20]  Michael A. Arbib,et al.  Action to Language via the Mirror Neuron System: Attention and the minimal subscene , 2006 .

[21]  C. Wernicke.  Der aphasische Symptomenkomplex , 1974 .

[22]  E E Smith,et al.  Components of verbal working memory: evidence from neuroimaging. , 1998, Proceedings of the National Academy of Sciences of the United States of America.

[23]  Francisco Aboitiz,et al.  Cortical memory mechanisms and language origins , 2006, Brain and Language.

[24]  G. Holmes,et al.  Sensory disturbances from cerebral lesions , 1911 .

[25]  Christine D. Wilson,et al.  Grounding conceptual knowledge in modality-specific systems , 2003, Trends in Cognitive Sciences.

[26]  Michael A. Arbib,et al.  From Schema Theory To Language , 1987 .

[27]  J. Piaget.  Biology and knowledge;: An essay on the relations between organic regulations and cognitive processes , 1971 .

[28]  Maria Mercedes Piñango.  Understanding the architecture of language: the possible role of neurology , 2006, Trends in Cognitive Sciences.

[29]  E. Miller,et al.  Memory fields of neurons in the primate prefrontal cortex. , 1998, Proceedings of the National Academy of Sciences of the United States of America.

[30]  Michael A. Arbib,et al.  Action to Language via the Mirror Neuron System: Neural homologies and the grounding of neurolinguistics , 2006 .

[31]  E. Bullmore,et al.  Common and Distinct Neural Substrates for Pragmatic, Semantic, and Syntactic Processing of Spoken Sentences: An fMRI Study , 2000, Journal of Cognitive Neuroscience.

[32]  Michael A. Arbib,et al.  Vision and Action in the Language-Ready Brain: From Mirror Neurons to SemRep , 2007, BVAI.

[33]  Bruce A. Draper,et al.  The schema system , 1988, International Journal of Computer Vision.

[34]  F. Aboitiz,et al.  The evolutionary origin of the language areas in the human brain. A neuroanatomical perspective , 1997, Brain Research Reviews.

[35]  B. Richmond,et al.  Role of perirhinal cortex in object perception, memory, and associations , 2001, Current Opinion in Neurobiology.

[36]  L. Barsalou,et al.  Whither structured representation? , 1999, Behavioral and Brain Sciences.

[37]  M. Arbib.  Aphasia, apraxia and the evolution of the language-ready brain , 2006 .

[38]  Michael A. Arbib,et al.  The Organization of Action-Oriented Memory for a Perceiving System. Part I: The Basic Model , 1971 .

[39]  N. Alpert,et al.  Localization of Syntactic Comprehension by Positron Emission Tomography , 1996, Brain and Language.

[40]  A. Goldman,et al.  Mirror neurons and the simulation theory of mind-reading , 1998, Trends in Cognitive Sciences.

[41]  Victor R. Lesser,et al.  The Hearsay-II Speech-Understanding System: Integrating Knowledge to Resolve Uncertainty , 1980, CSUR.

[42]  D. Poeppel,et al.  The cortical organization of speech processing , 2007, Nature Reviews Neuroscience.

[43]  Michael A. Arbib,et al.  Language evolution: neural homologies and neuroinformatics , 2003, Neural Networks.

[44]  Leslie G. Ungerleider,et al.  Neural correlates of category-specific knowledge , 1996, Nature.

[45]  David Marr,et al.  VISION A Computational Investigation into the Human Representation and Processing of Visual Information , 2009 .

[46]  Benjamin K. Bergen,et al.  Embodied Construction Grammar in Simulation-Based Language Understanding , 2003 .

[47]  E. Kaan,et al.  The brain circuitry of syntactic comprehension , 2002, Trends in Cognitive Sciences.

[48]  Michael A. Arbib,et al.  Sensorimotor Transformations in the Worlds of Frogs and Robots , 1995, Artif. Intell..

[49]  M. Arbib,et al.  Neural Organization: Structure, Function, and Dynamics , 1997 .

[50]  Luc Steels,et al.  Hierarchy in Fluid Construction Grammars , 2005, KI.

[51]  David Kemmerer,et al.  Action to Language via the Mirror Neuron System: Action verbs, argument structure constructions, and the mirror neuron system , 2006 .

[52]  D. Poeppel,et al.  Dorsal and ventral streams: a framework for understanding aspects of the functional anatomy of language , 2004, Cognition.

[53]  Peter Ford Dominey,et al.  Structure Mapping And Semantic Integration in a Construction-Based Neurolinguistic Model of Sentence Processing , 2006, Cortex.

[54]  Alex Martin,et al.  Representation of Manipulable Man-Made Objects in the Dorsal Stream , 2000, NeuroImage.

[55]  M. Arbib,et al.  Neurolinguistics must be computational , 1979 .

[56]  M. Arbib.  From monkey-like action recognition to human language: An evolutionary framework for neurolinguistics , 2005, Behavioral and Brain Sciences.

[57]  Michael A. Arbib,et al.  Perceptual Structures and Distributed Motor Control , 1981 .

[58]  M. Arbib.  Rana computatrix to human language: towards a computational neuroethology of language evolution , 2003, Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences.

[59]  Michael A. Arbib,et al.  Modeling parietal-premotor interactions in primate control of grasping , 1998, Neural Networks.

[60]  Michael A. Arbib,et al.  Schema design and implementation of the grasp-related mirror neuron system , 2002, Biological Cybernetics.

[61]  F. Pulvermüller.  Brain reflections of words and their meaning , 2001, Trends in Cognitive Sciences.

[62]  M. Arbib,et al.  Language within our grasp , 1998, Trends in Neurosciences.

[63]  Conrado A. Bosman,et al.  The Origin of Broca’s Area and Its Connections from an Ancestral Working Memory Network , 2006 .