Vision and Action in the Language-Ready Brain: From Mirror Neurons to SemRep

The general setting for our work is to locate language perception and production within the broader context of brain mechanisms for action and perception in general, modeling brain function in terms of the competition and cooperation of schemas. Particular emphasis is placed on mirror neurons - neurons active both for execution of a certain class of actions and for recognition of a (possibly broader) class of similar actions. We build on the early VISIONS model of schema-based computer analysis of static scenes to present SemRep, a graphical representation of dynamic visual scenes designed to support the generation of varied descriptions of episodes. Mechanisms for parsing and production of sentences are currently being implemented within Template Construction Grammar (TCG), a new form of construction grammar distinguished by its use of SemRep to express semantics.

[1]  L. Itti,et al.  Modeling the influence of task on attention , 2005, Vision Research.

[2]  Bruce A. Draper,et al.  The schema system , 1988, International Journal of Computer Vision.

[3]  Luc Steels,et al.  Hierarchy in Fluid Construction Grammars , 2005, KI.

[4]  M. Arbib.  From monkey-like action recognition to human language: An evolutionary framework for neurolinguistics , 2005, Behavioral and Brain Sciences.

[5]  Peter Ford Dominey,et al.  Structure Mapping And Semantic Integration in a Construction-Based Neurolinguistic Model of Sentence Processing , 2006, Cortex.

[6]  Michael A. Arbib,et al.  Affordances. Motivations, and the World Graph Theory , 1998, Adapt. Behav..

[7]  Victor R. Lesser,et al.  The Hearsay-II Speech-Understanding System: Integrating Knowledge to Resolve Uncertainty , 1980, CSUR.

[8]  Michael A. Arbib,et al.  Extending the mirror neuron system model, I , 2007, Biological Cybernetics.

[9]  Lisa Gershkoff-Stowe,et al.  Is there a natural order for expressing semantic relations? , 2002, Cognitive Psychology.

[10]  Michael A. Arbib,et al.  Perceptual Structures and Distributed Motor Control , 1981 .

[11]  M. Arbib.  Rana computatrix to human language: towards a computational neuroethology of language evolution , 2003, Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences.

[12]  M. Arbib.  Action to language via the mirror neuron system , 2006 .

[13]  J. Feldman,et al.  Embodied meaning in a neural theory of language , 2004, Brain and Language.

[14]  K. Amunts,et al.  Broca's region , 2006 .

[15]  Luc Steels,et al.  Perspective alignment in spatial language , 2006, Spatial Language and Dialogue.

[16]  M. Arbib.  Broca’s Area in System Perspective: Language in the Context of Action-Oriented Perception , 2006 .

[17]  Michael A. Arbib,et al.  Modeling parietal-premotor interactions in primate control of grasping , 1998, Neural Networks.

[18]  Michael A. Arbib,et al.  Sensorimotor Transformations in the Worlds of Frogs and Robots , 1995, Artif. Intell..

[19]  Adele E. Goldberg.  Constructions: a new theoretical approach to language , 2003, Trends in Cognitive Sciences.

[20]  Michael A. Arbib,et al.  Action to Language via the Mirror Neuron System: Attention and the minimal subscene , 2006 .

[21]  M. Arbib,et al.  Neural expectations : A possible evolutionary path from manual skills to language , 1996 .

[22]  L. Steels.  Evolving grounded communication for robots , 2003, Trends in Cognitive Sciences.

[23]  Ricardo Baeza-Yates,et al.  Computer Science 2 , 1994 .

[24]  Noam Chomsky,et al.  Lectures on Government and Binding , 1981 .

[25]  Deb Roy,et al.  Semiotic schemas: A framework for grounding language in action and perception , 2005, Artif. Intell..

[26]  A. Goldman,et al.  Mirror neurons and the simulation theory of mind-reading , 1998, Trends in Cognitive Sciences.

[27]  C. Fillmore,et al.  Regularity and Idiomaticity in Grammatical Constructions: The Case of Let Alone , 1988 .

[28]  J. Gibson.  The Ecological Approach to Visual Perception , 1979 .

[29]  Ulrich Furbach.  KI 2005: Advances in Artificial Intelligence , 2005 .

[30]  BART KOSKO,et al.  Bidirectional associative memories , 1988, IEEE Trans. Syst. Man Cybern..

[31]  F. D. Saussure.  Cours de linguistique générale , 1924 .

[32]  Angelo Cangelosi,et al.  An Embodied Model for Sensorimotor Grounding and Grounding Transfer: Experiments With Epigenetic Robots , 2006, Cogn. Sci..

[33]  Michael A. Arbib,et al.  Schema design and implementation of the grasp-related mirror neuron system , 2002, Biological Cybernetics.