Conceptual grounding of language in action and perception: a neurocomputational model of the emergence of category specificity and semantic hubs

Current neurobiological accounts of language and cognition offer diverging views on the questions of ‘where’ and ‘how’ semantic information is stored and processed in the human brain. Neuroimaging data showing consistent activation of different multi‐modal areas during word and sentence comprehension suggest that all meanings are processed indistinctively, by a set of general semantic centres or ‘hubs’. However, words belonging to specific semantic categories selectively activate modality‐preferential areas; for example, action‐related words spark activity in dorsal motor cortex, whereas object‐related ones activate ventral visual areas. The evidence for category‐specific and category‐general semantic areas begs for a unifying explanation, able to integrate the emergence of both. Here, a neurobiological model offering such an explanation is described. Using a neural architecture replicating anatomical and neurophysiological features of frontal, occipital and temporal cortices, basic aspects of word learning and semantic grounding in action and perception were simulated. As the network underwent training, distributed lexico‐semantic circuits spontaneously emerged. These circuits exhibited different cortical distributions that reached into dorsal‐motor or ventral‐visual areas, reflecting the correlated category‐specific sensorimotor patterns that co‐occurred during action‐ or object‐related semantic grounding, respectively. Crucially, substantial numbers of neurons of both types of distributed circuits emerged in areas interfacing between modality‐preferential regions, i.e. in multimodal connection hubs, which therefore became loci of general semantic binding. By relating neuroanatomical structure and cellular‐level learning mechanisms with system‐level cognitive function, this model offers a neurobiological account of category‐general and category‐specific semantic areas based on the different cortical distributions of the underlying semantic circuits.

[1]  S C Kleene,et al.  Representation of Events in Nerve Nets and Finite Automata , 1951 .

[2]  J. Fodor,et al.  The structure of a semantic theory , 1963 .

[3]  J. Cowan,et al.  Excitatory and inhibitory interactions in localized populations of model neurons. , 1972, Biophysical journal.

[4]  Allan Collins,et al.  A spreading-activation theory of semantic processing , 1975 .

[5]  Braitenberg,et al.  Cortical architectonics: General and areal , 1978 .

[6]  John R. Searle,et al.  Minds, brains, and programs , 1980, Behavioral and Brain Sciences.

[7]  O. Andy The prefrontal cortex: Anatomy, physiology and neuropsychology of the frontal lobe , 1981 .

[8]  Leslie G. Ungerleider Two cortical visual systems , 1982 .

[9]  G. Palm Neural Assemblies , 1982, Studies of Brain Function.

[10]  Leslie G. Ungerleider,et al.  Object vision and spatial vision: two cortical pathways , 1983, Trends in Neurosciences.

[11]  T. Shallice,et al.  Category specific semantic impairments , 1984 .

[12]  D. Pandya,et al.  Architecture and Connections of Cortical Association Areas , 1985 .

[13]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[14]  Geoffrey E. Hinton,et al.  Learning representations by back-propagation errors, nature , 1986 .

[15]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[16]  E. Warrington,et al.  Categories of knowledge. Further fractionations and an attempted integration. , 1987, Brain : a journal of neurology.

[17]  K Watanabe,et al.  Connections of area 8 with area 6 in the brain of the macaque monkey , 1988, The Journal of comparative neurology.

[18]  Richard J. Brown Neuropsychology Mental Structure , 1989 .

[19]  Stevan Harnad,et al.  Symbol grounding problem , 1990, Scholarpedia.

[20]  W. Singer,et al.  Different voltage-dependent thresholds for inducing long-term depression and long-term potentiation in slices of rat visual cortex , 1990, Nature.

[21]  Michael I. Jordan,et al.  A more biologically plausible learning rule for neural networks. , 1991, Proceedings of the National Academy of Sciences of the United States of America.

[22]  D. J. Felleman,et al.  Distributed hierarchical processing in the primate cerebral cortex. , 1991, Cerebral cortex.

[23]  E. Capaldi,et al.  The organization of behavior. , 1992, Journal of applied behavior analysis.

[24]  M. Tomasello,et al.  Joint attention on actions: acquiring verbs in ostensive and non-ostensive contexts , 1992, Journal of Child Language.

[25]  W. Singer,et al.  Long-term depression of excitatory synaptic transmission and its relationship to long-term potentiation , 1993, Trends in Neurosciences.

[26]  Leslie G. Ungerleider,et al.  The modular organization of projections from areas V1 and V2 to areas V4 and TEO in macaques , 1993, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[27]  R. Malach,et al.  Cortical hierarchy reflected in the organization of intrinsic connections in macaque monkey visual cortex , 1993, The Journal of comparative neurology.

[28]  Leslie G. Ungerleider,et al.  Cortical connections of inferior temporal area TEO in macaque monkeys , 1993, The Journal of comparative neurology.

[29]  Malcolm P. Young,,et al.  Analysis of Connectivity: Neural Systems in the Cerebral Cortex , 1994, Reviews in the neurosciences.

[30]  J. B. Preston,et al.  Interconnections between the prefrontal cortex and the premotor areas in the frontal lobe , 1994, The Journal of comparative neurology.

[31]  G. Kane Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol 1: Foundations, vol 2: Psychological and Biological Models , 1994 .

[32]  Leslie G. Ungerleider,et al.  Connections of inferior temporal areas TEO and TE with parietal and frontal cortex in macaque monkeys. , 1994, Cerebral cortex.

[33]  Leslie G. Ungerleider,et al.  ‘What’ and ‘where’ in the human brain , 1994, Current Opinion in Neurobiology.

[34]  D. Pandya,et al.  Anatomy of the auditory cortex. , 1995, Revue neurologique.

[35]  Gully A. P. C. Burns,et al.  The Analysis of Cortical Connectivity , 1995 .

[36]  M. Arbib,et al.  Grasping objects: the cortical mechanisms of visuomotor transformation , 1995, Trends in Neurosciences.

[37]  Michael A. Arbib,et al.  The handbook of brain theory and neural networks , 1995, A Bradford book.

[38]  A. Damasio,et al.  A neural basis for lexical retrieval , 1996, Nature.

[39]  J. Elman,et al.  Rethinking Innateness: A Connectionist Perspective on Development , 1996 .

[40]  Leslie G. Ungerleider,et al.  Neural correlates of category-specific knowledge , 1996, Nature.

[41]  T. Landauer,et al.  A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge. , 1997 .

[42]  Kim Plunkett,et al.  Theories of early language acquisition , 1997, Trends in Cognitive Sciences.

[43]  Jon H Kaas,et al.  Topographic Maps are Fundamental to Sensory Processing , 1997, Brain Research Bulletin.

[44]  Alan L. Yuille,et al.  Winner-take-all mechanisms , 1998 .

[45]  R. O’Reilly Six principles for biologically based computational models of cortical cognition , 1998, Trends in Cognitive Sciences.

[46]  Prof. Dr. Dr. Valentino Braitenberg,et al.  Cortex: Statistics and Geometry of Neuronal Connectivity , 1998, Springer Berlin Heidelberg.

[47]  M. Mishkin,et al.  Dual streams of auditory afferents target multiple domains in the primate prefrontal cortex , 1999, Nature Neuroscience.

[48]  G. Lakoff,et al.  Philosophy in the flesh : the embodied mind and its challenge to Western thought , 1999 .

[49]  E. Yeterian,et al.  MRI-Based Topographic Parcellation of Human Cerebral White Matter and Nuclei II. Rationale and Applications with Systematics of Cerebral Connectivity , 1999, NeuroImage.

[50]  F. Pulvermüller Words in the brain's language , 1999, Behavioral and Brain Sciences.

[51]  Gary S. Dell,et al.  Connectionist models of language production: lexical access and grammatical encoding , 1999, Cogn. Sci..

[52]  L. Barsalou,et al.  Whither structured representation? , 1999, Behavioral and Brain Sciences.

[53]  F Pulvermüller,et al.  Nouns and verbs in the intact brain: evidence from event-related potentials and high-frequency cortical responses. , 1999, Cerebral cortex.

[54]  P. Goldman-Rakic,et al.  Auditory belt and parabelt projections to the prefrontal cortex in the Rhesus monkey , 1999, The Journal of comparative neurology.

[55]  J. Eggert,et al.  Unifying framework for neuronal assembly dynamics. , 2000, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[56]  David C. Plaut,et al.  Are non-semantic morphological effects incompatible with a distributed connectionist approach to lexical processing? , 2000 .

[57]  C. Price The anatomy of language: contributions from functional neuroimaging , 2000, Journal of anatomy.

[58]  J. Donoghue,et al.  Learning-induced LTP in neocortex. , 2000, Science.

[59]  J. Kaas,et al.  Subdivisions of auditory cortex and processing streams in primates. , 2000, Proceedings of the National Academy of Sciences of the United States of America.

[60]  J. Rauschecker,et al.  Mechanisms and streams for processing of "what" and "where" in auditory cortex. , 2000, Proceedings of the National Academy of Sciences of the United States of America.

[61]  G. Matthews,et al.  Neurobiology: Molecules, Cells and Systems , 2001 .

[62]  Markus Kiefer,et al.  The limits of a distributed account of conceptual knowledge , 2001, Trends in Cognitive Sciences.

[63]  J. Grady Philosophy in the Flesh: The Embodied Mind and its Challenge to Western Thought , 2002 .

[64]  D. Pandya,et al.  Comparative cytoarchitectonic analysis of the human and the macaque ventrolateral prefrontal cortex and corticocortical connection patterns in the monkey , 2002, The European journal of neuroscience.

[65]  P. Strick,et al.  Motor areas in the frontal lobe of the primate , 2002, Physiology & Behavior.

[66]  S. Bookheimer Functional MRI of language: new approaches to understanding the cortical organization of semantic processing. , 2002, Annual review of neuroscience.

[67]  P. Matthews,et al.  Semantic Processing in the Left Inferior Prefrontal Cortex: A Combined Functional Magnetic Resonance Imaging and Transcranial Magnetic Stimulation Study , 2003, Journal of Cognitive Neuroscience.

[68]  Leslie G. Ungerleider,et al.  Projections from inferior temporal cortex to prefrontal cortex via the uncinate fascicle in rhesus monkeys , 2004, Experimental Brain Research.

[69]  R. Douglas,et al.  Neuronal circuits of the neocortex. , 2004, Annual review of neuroscience.

[70]  M. Bear,et al.  LTP and LTD An Embarrassment of Riches , 2004, Neuron.

[71]  F. T. Husain,et al.  Relating neuronal dynamics for auditory object processing to neuroimaging activity: a computational modeling and an fMRI study , 2004, NeuroImage.

[72]  I. Johnsrude,et al.  Somatotopic Representation of Action Words in Human Motor and Premotor Cortex , 2004, Neuron.

[73]  Ronald,et al.  Learning representations by backpropagating errors , 2004 .

[74]  Joyce P. Griffin-Sobel,et al.  "How do you know that?". , 2004, Clinical journal of oncology nursing.

[75]  G. Rizzolatti,et al.  The mirror-neuron system. , 2004, Annual review of neuroscience.

[76]  Matthew A. Lambon Ralph,et al.  Lateralization of ventral and dorsal auditory-language pathways in the human brain , 2005, NeuroImage.

[77]  Vittorio Gallese,et al.  Listening to Action-related Sentences Activates Fronto-parietal Motor Circuits , 2005, Journal of Cognitive Neuroscience.

[78]  P. Strick,et al.  Frontal Lobe Inputs to the Digit Representations of the Motor Areas on the Lateral Surface of the Hemisphere , 2005, The Journal of Neuroscience.

[79]  Derek K. Jones,et al.  Perisylvian language networks of the human brain , 2005, Annals of neurology.

[80]  G. Rizzolatti,et al.  Congruent Embodied Representations for Visually Presented Actions and Linguistic Phrases Describing Actions , 2006, Current Biology.

[81]  J. Duncan EPS Mid-Career Award 2004: Brain mechanisms of attention , 2006, Quarterly journal of experimental psychology.

[82]  Satrajit S. Ghosh,et al.  Neural modeling and imaging of the cortical interactions underlying syllable production , 2006, Brain and Language.

[83]  Brian B. Avants,et al.  Lagrangian frame diffeomorphic image registration: Morphometric comparison of human and chimpanzee cortex , 2006, Medical Image Anal..

[84]  René Doursat,et al.  Neocortical Self-Structuration as a Basis for Learning , 2006 .

[85]  Namir,et al.  Authors , 1947, Praxis der Kinderpsychologie und Kinderpsychiatrie.

[86]  F. Pulvermüller,et al.  Category-specific conceptual processing of color and form in left fronto-temporal cortex. , 2006, Cerebral cortex.

[87]  Bernard Mazoyer,et al.  Meta-analyzing left hemisphere language areas: Phonology, semantics, and sentence processing , 2006, NeuroImage.

[88]  S. Harnad Symbol grounding problem , 1991, Scholarpedia.

[89]  D. V. Essen,et al.  Surface-Based and Probabilistic Atlases of Primate Cerebral Cortex , 2007, Neuron.

[90]  W. K. Simmons,et al.  A common neural substrate for perceiving and knowing about color , 2007, Neuropsychologia.

[91]  Justin L. Vincent,et al.  Intrinsic functional architecture in the anaesthetized monkey brain , 2007, Nature.

[92]  B. Mesquita,et al.  Adjustment to Chronic Diseases and Terminal Illness Health Psychology : Psychological Adjustment to Chronic Disease , 2006 .

[93]  D. Pandya,et al.  Association fibre pathways of the brain: parallel observations from diffusion spectrum imaging and autoradiography. , 2007, Brain : a journal of neurology.

[94]  T. Rogers,et al.  Where do you know what you know? The representation of semantic knowledge in the human brain , 2007, Nature Reviews Neuroscience.

[95]  Friedemann Pulvermüller,et al.  A neuronal model of the language cortex , 2007, Neurocomputing.

[96]  Lizabeth M Romanski,et al.  Representation and integration of auditory and visual stimuli in the primate ventral lateral prefrontal cortex. , 2007, Cerebral cortex.

[97]  D. Pandya,et al.  The extreme capsule in humans and rethinking of the language circuitry , 2009, Brain Structure and Function.

[98]  L. Barsalou Grounded cognition. , 2008, Annual review of psychology.

[99]  Volkmar Glauche,et al.  Ventral and dorsal pathways for language , 2008, Proceedings of the National Academy of Sciences.

[100]  Timothy E. J. Behrens,et al.  The evolution of the arcuate fasciculus revealed with comparative DTI , 2008, Nature Neuroscience.

[101]  Friedemann Pulvermüller,et al.  A neuroanatomically grounded Hebbian-learning model of attention–language interactions in the human brain , 2008, The European journal of neuroscience.

[102]  Friedemann Pulvermüller,et al.  Effects of Attention on what is known and what is not: MEG Evidence for Functionally Discrete Memory Circuits , 2008, Front. Hum. Neurosci..

[103]  D. Pandya,et al.  Distinct Parietal and Temporal Pathways to the Homologues of Broca's Area in the Monkey , 2009, PLoS biology.

[104]  Ferath Kherif,et al.  Distributed cell assemblies for general lexical and category‐specific semantic processing as revealed by fMRI cluster analysis , 2009, Human brain mapping.

[105]  J. Rauschecker,et al.  Maps and streams in the auditory cortex: nonhuman primates illuminate human speech processing , 2009, Nature Neuroscience.

[106]  Friedemann Pulvermüller,et al.  Recruitment and Consolidation of Cell Assemblies for Words by Way of Hebbian Learning and Competition in a Multi-Layer Neural Network , 2009, Cognitive Computation.

[107]  J. Werker,et al.  Infants' learning of novel words in a stochastic environment. , 2009, Developmental psychology.

[108]  William W. Graves,et al.  Where is the semantic system? A critical review and meta-analysis of 120 functional neuroimaging studies. , 2009, Cerebral cortex.

[109]  M. Dawkins The Noisy Brain: Stochastic Dynamics as a Principle of Brain Function The Noisy Brain: Stochastic Dynamics as a Principle of Brain Function. By Edmund T. Rolls & Gustavo Deco. Oxford: Oxford University Press (2010). Pp. 310. Price £37.95 hardback. , 2010, Animal Behaviour.

[110]  D. Kemmerer,et al.  The Two-Level Theory of verb meaning: An approach to integrating the semantics of action with the mirror neuron system , 2010, Brain and Language.

[111]  L. Fadiga,et al.  Active perception: sensorimotor circuits as a cortical basis for language , 2010, Nature Reviews Neuroscience.

[112]  Rutvik H. Desai,et al.  The neurobiology of semantic memory , 2011, Trends in Cognitive Sciences.

[113]  M. Graziano,et al.  Human consciousness and its relationship to social neuroscience: A novel hypothesis , 2011, Cognitive neuroscience.

[114]  Friedemann Pulvermüller,et al.  From sounds to words: A neurocomputational model of adaptation, inhibition and memory processes in auditory change detection , 2011, NeuroImage.

[115]  T. Rogers,et al.  Lichtheim 2: Synthesizing Aphasia and the Neural Basis of Language in a Neurocomputational Model of the Dual Dorsal-Ventral Language Pathways , 2011, Neuron.

[116]  F. Pulvermüller,et al.  Reading salt activates gustatory brain regions: fMRI evidence for semantic grounding in a novel sensory modality. , 2012, Cerebral cortex.

[117]  B. Bahrami,et al.  Coming of age: A review of embodiment and the neuroscience of semantics , 2012, Cortex.

[118]  Deepak N. Pandya,et al.  The prefrontal cortex: Comparative architectonic organization in the human and the macaque monkey brains , 2012, Cortex.

[119]  C Sergio Espinoza,et al.  Principles of neural sciences , 2012 .

[120]  Friedemann Pulvermüller,et al.  Body-part-specific Representations of Semantic Noun Categories , 2012, Journal of Cognitive Neuroscience.

[121]  D. Tranel,et al.  Behavioral patterns and lesion sites associated with impaired processing of lexical and conceptual knowledge of actions , 2012, Cortex.

[122]  K. Nader,et al.  The role of metaplasticity mechanisms in regulating memory destabilization and reconsolidation , 2012, Neuroscience & Biobehavioral Reviews.

[123]  M. Kiefer,et al.  Conceptual representations in mind and brain: Theoretical developments, current evidence and future directions , 2012, Cortex.

[124]  A. Glenberg,et al.  Action-based language: A theory of language acquisition, comprehension, and production , 2012, Cortex.

[125]  O. Sporns,et al.  Network hubs in the human brain , 2013, Trends in Cognitive Sciences.

[126]  Gustavo Deco,et al.  Resting brains never rest: computational insights into potential cognitive architectures , 2013, Trends in Neurosciences.

[127]  F. Pulvermüller How neurons make meaning: brain mechanisms for embodied and abstract-symbolic semantics , 2013, Trends in Cognitive Sciences.

[128]  Friedemann Pulvermüller,et al.  Neuronal correlates of decisions to speak and act: Spontaneous emergence and dynamic topographies in a computational model of frontal and temporal areas , 2013, Brain and Language.

[129]  A. Young,et al.  Human Cognitive Neuropsychology , 2013 .

[130]  E. Rolls,et al.  Brain mechanisms for perceptual and reward-related decision-making , 2013, Progress in Neurobiology.

[131]  Friedemann Pulvermüller,et al.  Sensorimotor semantics on the spot: brain activity dissociates between conceptual categories within 150 ms , 2013, Scientific Reports.

[132]  Y. Shtyrov,et al.  Automatic ultrarapid activation and inhibition of cortical motor systems in spoken word comprehension , 2014, Proceedings of the National Academy of Sciences.

[133]  Pascale Tremblay,et al.  The Language Connectome , 2014, The Neuroscientist : a review journal bringing neurobiology, neurology and psychiatry.

[134]  Günther Palm,et al.  Cell assemblies in the cerebral cortex , 2014, Biological Cybernetics.

[135]  Friedemann Pulvermüller,et al.  From sensorimotor learning to memory cells in prefrontal and temporal association cortex: A neurocomputational study of disembodiment , 2014, Cortex.

[136]  James K. Rilling,et al.  Comparative primate neuroimaging: insights into human brain evolution , 2014, Trends in Cognitive Sciences.

[137]  D. Kemmerer Are the motor features of verb meanings represented in the precentral motor cortices? Yes, but within the context of a flexible, multilevel architecture for conceptual knowledge , 2015, Psychonomic Bulletin & Review.

[138]  Claire Colebrook,et al.  What Is It Like to Be a Human , 2015 .