Supplementary Information for Natural Speech Reveals the Semantic Maps That Tile Human Cerebral Cortex

The meaning of language is represented in regions of the cerebral cortex collectively known as the ‘semantic system’. However, little of the semantic system has been mapped comprehensively, and the semantic selectivity of most regions is unknown. Here we systematically map semantic selectivity across the cortex using voxel-wise modelling of functional MRI (fMRI) data collected while subjects listened to hours of narrative stories. We show that the semantic system is organized into intricate patterns that seem to be consistent across individuals. We then use a novel generative model to create a detailed semantic atlas. Our results suggest that most areas within the semantic system represent information about specific semantic domains, or groups of related concepts, and our atlas shows which domains are represented in each area. This study demonstrates that data-driven methods—commonplace in studies of human neuroanatomy and functional connectivity—provide a powerful and efficient means for mapping functional representations in the brain.

[1]  H. Kaiser The varimax criterion for analytic rotation in factor analysis , 1958 .

[2]  R. C. Oldfield The assessment and analysis of handedness: the Edinburgh inventory. , 1971, Neuropsychologia.

[3]  E. Warrington Quarterly Journal of Experimental Psychology the Selective Impairment of Semantic Memory the Selective Impairment of Semantic Memory , 2022 .

[4]  Geoffrey E. Hinton,et al.  A Learning Algorithm for Boltzmann Machines , 1985, Cogn. Sci..

[5]  T. Landauer,et al.  Indexing by Latent Semantic Analysis , 1990 .

[6]  H. Lüders,et al.  Basal temporal language area. , 1991, Brain : a journal of neurology.

[7]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[8]  Richard D. Hichwa,et al.  A neural basis for lexical retrieval , 1996, Nature.

[9]  Curt Burgess,et al.  Producing high-dimensional semantic spaces from lexical co-occurrence , 1996 .

[10]  N. Kanwisher,et al.  The Fusiform Face Area: A Module in Human Extrastriate Cortex Specialized for Face Perception , 1997, The Journal of Neuroscience.

[11]  Nancy Kanwisher,et al.  A cortical representation of the local visual environment , 1998, Nature.

[12]  A. Caramazza,et al.  Domain-Specific Knowledge Systems in the Brain: The Animate-Inanimate Distinction , 1998, Journal of Cognitive Neuroscience.

[13]  Irene P. Kan,et al.  Verb generation in patients with focal frontal lesions: a neuropsychological test of neuroimaging findings. , 1998, Proceedings of the National Academy of Sciences of the United States of America.

[14]  L. Cipolotti,et al.  Dynamic aphasia: an inability to select between competing verbal responses? , 1998, Brain : a journal of neurology.

[15]  C. Price,et al.  Functional Neuroanatomy of the Semantic System: Divisible by What? , 1998, Journal of Cognitive Neuroscience.

[16]  A. Shimamura,et al.  Letter and category fluency in patients with frontal lobe lesions. , 1998, Neuropsychology.

[17]  Leslie G. Ungerleider,et al.  An area specialized for spatial working memory in human frontal cortex. , 1998, Science.

[18]  Anders M. Dale,et al.  Cortical Surface-Based Analysis I. Segmentation and Surface Reconstruction , 1999, NeuroImage.

[19]  K. Amunts,et al.  Brodmann's Areas 17 and 18 Brought into Stereotaxic Space—Where and How Variable? , 2000, NeuroImage.

[20]  Arthur E. Hoerl,et al.  Ridge Regression: Biased Estimation for Nonorthogonal Problems , 2000, Technometrics.

[21]  P. Goldman-Rakic,et al.  Segregation of working memory functions within the dorsolateral prefrontal cortex , 2000, Experimental Brain Research.

[22]  E. Ringelstein,et al.  Handedness and hemispheric language dominance in healthy humans. , 2000, Brain : a journal of neurology.

[23]  T. Schormann,et al.  Functional delineation of the human occipito-temporal areas related to face and scene processing. A PET study. , 2000, Brain : a journal of neurology.

[24]  D Y von Cramon,et al.  Segregating semantic and syntactic aspects of processing in the human brain: an fMRI investigation of different word types. , 2000, Cerebral cortex.

[25]  Eric Jones,et al.  SciPy: Open Source Scientific Tools for Python , 2001 .

[26]  N. Kanwisher,et al.  A Cortical Area Selective for Visual Processing of the Human Body , 2001, Science.

[27]  G L Shulman,et al.  INAUGURAL ARTICLE by a Recently Elected Academy Member:A default mode of brain function , 2001 .

[28]  D. Bowers,et al.  Verbal Encoding Deficits in a Patient with a Left Retrosplenial Lesion , 2001, Neurocase.

[29]  Stephen M. Smith,et al.  A global optimisation method for robust affine registration of brain images , 2001, Medical Image Anal..

[30]  Maneesh Sahani,et al.  How Linear are Auditory Cortical Responses? , 2002, NIPS.

[31]  Geoffrey E. Hinton Training Products of Experts by Minimizing Contrastive Divergence , 2002, Neural Computation.

[32]  Paul Boersma,et al.  Praat, a system for doing phonetics by computer , 2002 .

[33]  S. Bookheimer Functional MRI of language: new approaches to understanding the cortical organization of semantic processing. , 2002, Annual review of neuroscience.

[34]  Jesper Andersson,et al.  Isolating the retrieval of imagined pictures during episodic memory: activation of the left precuneus and left prefrontal cortex , 2003, NeuroImage.

[35]  R Saxe,et al.  People thinking about thinking people The role of the temporo-parietal junction in “theory of mind” , 2003, NeuroImage.

[36]  Paul Boersma,et al.  Praat: doing phonetics by computer , 2003 .

[37]  Uta Noppeney,et al.  Retrieval of abstract semantics , 2004, NeuroImage.

[38]  Alexander Borst,et al.  Quantifying variability in neural responses and its application for the validation of model predictions , 2004, Network.

[39]  Istvan Molnar-Szakacs,et al.  Watching social interactions produces dorsomedial prefrontal and medial parietal BOLD fMRI signal increases compared to a resting baseline , 2004, NeuroImage.

[40]  J. Gallant,et al.  Predicting neuronal responses during natural vision , 2005, Network.

[41]  Benjamin J. Shannon,et al.  Parietal lobe contributions to episodic memory retrieval , 2005, Trends in Cognitive Sciences.

[42]  M. Sigman,et al.  Opinion TRENDS in Cognitive Sciences Vol.9 No.7 July 2005 The neural code for written words: a proposal , 2022 .

[43]  David A. Medler,et al.  Distinct Brain Systems for Processing Concrete and Abstract Concepts , 2005, Journal of Cognitive Neuroscience.

[44]  A. Caramazza,et al.  The organisation of conceptual knowledge in the brain: The future's past and some future directions , 2006, Cognitive neuropsychology.

[45]  M. Riesenhuber Appearance Isn't Everything: News on Object Representation in Cortex , 2007, Neuron.

[46]  D. Poeppel,et al.  The cortical organization of speech processing , 2007, Nature Reviews Neuroscience.

[47]  D. Schacter,et al.  The Brain's Default Network , 2008, Annals of the New York Academy of Sciences.

[48]  A. Caramazza,et al.  Concepts Are More than Percepts: The Case of Action Verbs , 2008, The Journal of Neuroscience.

[49]  Damien A. Fair,et al.  Defining functional areas in individual human brains using resting functional connectivity MRI , 2008, NeuroImage.

[50]  Mark Liberman,et al.  Speaker identification on the SCOTUS corpus , 2008 .

[51]  N. Kanwisher,et al.  Interpreting fMRI data: maps, modules and dimensions , 2008, Nature Reviews Neuroscience.

[52]  Tom Michael Mitchell,et al.  Predicting Human Brain Activity Associated with the Meanings of Nouns , 2008, Science.

[53]  Ryan J. Prenger,et al.  Bayesian Reconstruction of Natural Images from Human Brain Activity , 2009, Neuron.

[54]  William W. Graves,et al.  Where is the semantic system? A critical review and meta-analysis of 120 functional neuroimaging studies. , 2009, Cerebral cortex.

[55]  Elizabeth Jefferies,et al.  Semantic Processing in the Anterior Temporal Lobes: A Meta-analysis of the Functional Neuroimaging Literature , 2010, Journal of Cognitive Neuroscience.

[56]  Patrick Pantel,et al.  From Frequency to Meaning: Vector Space Models of Semantics , 2010, J. Artif. Intell. Res..

[57]  Tom Michael Mitchell,et al.  A Neurosemantic Theory of Concrete Noun Representation Based on the Underlying Brain Codes , 2010, PloS one.

[58]  N. Kanwisher,et al.  New method for fMRI investigations of language: defining ROIs functionally in individual subjects. , 2010, Journal of neurophysiology.

[59]  C. Price The anatomy of language: a review of 100 fMRI studies published in 2009 , 2010, Annals of the New York Academy of Sciences.

[60]  Rutvik H. Desai,et al.  The neurobiology of semantic memory , 2011, Trends in Cognitive Sciences.

[61]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[62]  C. Honey,et al.  Topographic Mapping of a Hierarchy of Temporal Receptive Windows Using a Narrated Story , 2011, The Journal of Neuroscience.

[63]  Timothy D. Verstynen,et al.  Using pulse oximetry to account for high and low frequency physiological artifacts in the BOLD signal , 2011, NeuroImage.

[64]  J. Gallant,et al.  Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies , 2011, Current Biology.

[65]  P. Downing,et al.  Mapping brain activation and information during category-specific visual working memory. , 2012, Journal of neurophysiology.

[66]  J. Rauschecker,et al.  Phoneme and word recognition in the auditory ventral stream , 2012, Proceedings of the National Academy of Sciences.

[67]  Alfonso Caramazza,et al.  Independent Representations of Verbs and Actions in Left Lateral Temporal Cortex , 2012, Journal of Cognitive Neuroscience.

[68]  Jack L. Gallant,et al.  A Continuous Semantic Space Describes the Representation of Thousands of Object and Action Categories across the Human Brain , 2012, Neuron.

[69]  A. Schleicher,et al.  Organization of the Human Inferior Parietal Lobule Based on Receptor Architectonics , 2012, Cerebral cortex.

[70]  Geoffrey Zweig,et al.  Linguistic Regularities in Continuous Space Word Representations , 2013, NAACL.

[71]  Brian Murphy,et al.  Simultaneously Uncovering the Patterns of Brain Regions Involved in Different Story Reading Subprocesses , 2014, PloS one.

[72]  Keith Johnson,et al.  Phonetic Feature Encoding in Human Superior Temporal Gyrus , 2014, Science.

[73]  Travis E. Oliphant,et al.  Guide to NumPy , 2015 .

[74]  Jack L. Gallant,et al.  Pycortex: an interactive surface visualizer for fMRI , 2015, Front. Neuroinform..

[75]  Alexander G. Huth,et al.  PrAGMATiC: a Probabilistic and Generative Model of Areas Tiling the Cortex , 2015, 1504.03622.