The representational space of observed actions

Categorizing and understanding other people’s actions is a key human capability. Whereas there exists a growing literature regarding the organization of objects, the representational space underlying the organization of observed actions remains largely unexplored. Here we examined the organizing principles of a large set of actions and the corresponding neural representations. Using multiple regression representational similarity analysis of fMRI data, in which we accounted for variability due to major action components (body parts, scenes, movements, objects, sociality, transitivity) and three control models (distance between observer and actor, number of people, HMAX-C1), we found that the semantic dissimilarity structure was best captured by patterns of activation in the lateral occipitotemporal cortex (LOTC). Together, our results demonstrate that the organization of observed actions in the LOTC resembles the organizing principles used by participants to classify actions behaviorally, in line with the view that this region is crucial for accessing the meaning of actions.

[1]  J. A. Hartigan,et al.  A k-means clustering algorithm , 1979 .

[2]  L. Talmy Lexicalisation patterns: semantic structure in lexical forms , 1985 .

[3]  P. Rousseeuw Silhouettes: a graphical aid to the interpretation and validation of cluster analysis , 1987 .

[4]  D H Brainard,et al.  The Psychophysics Toolbox. , 1997, Spatial vision.

[5]  A. Caramazza,et al.  Domain-Specific Knowledge Systems in the Brain: The Animate-Inanimate Distinction , 1998, Journal of Cognitive Neuroscience.

[6]  J. Haxby,et al.  Attribute-based neural substrates in temporal cortex for perceiving and knowing about objects , 1999, Nature Neuroscience.

[7]  T. Poggio,et al.  Hierarchical models of object recognition in cortex , 1999, Nature Neuroscience.

[8]  A. Dale,et al.  Cortical Surface-Based Analysis II: Inflation, Flattening, and a Surface-Based Coordinate System , 1999, NeuroImage.

[9]  N. Kanwisher,et al.  The Human Body , 2001 .

[10]  A. Ishai,et al.  Distributed and Overlapping Representations of Faces and Objects in Ventral Temporal Cortex , 2001, Science.

[11]  D. V. von Cramon,et al.  Separable neuronal circuitries for manipulable and non-manipulable objects in working memory. , 2002, Cerebral cortex.

[12]  Robert L. Mason,et al.  Statistical Design and Analysis of Experiments , 2003 .

[13]  G. Rizzolatti,et al.  Two different streams form the dorsal visual system: anatomy and functions , 2003, Experimental Brain Research.

[14]  M. Corbetta,et al.  Extrastriate body area in human occipital cortex responds to the performance of motor actions , 2004, Nature Neuroscience.

[15]  P. Downing,et al.  Selectivity for the human body in the fusiform gyrus. , 2005, Journal of neurophysiology.

[16]  Alison J. Wiggett,et al.  Patterns of fMRI Activity Dissociate Overlapping Functional Brain Areas that Respond to Biological Motion , 2006, Neuron.

[17]  N. Kanwisher,et al.  Domain specificity in visual cortex. , 2006, Cerebral cortex.

[18]  Rainer Goebel,et al.  Information-based functional brain mapping. , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[19]  Kenneth F. Valyear,et al.  Human parietal cortex in action , 2006, Current Opinion in Neurobiology.

[20]  Alison J. Wiggett,et al.  Behavioral / Systems / Cognitive Functional Magnetic Resonance Imaging Investigation of Overlapping Lateral Occipitotemporal Activations Using Multi-Voxel Pattern Analysis , 2006 .

[21]  Thomas Serre,et al.  A feedforward architecture accounts for rapid categorization , 2007, Proceedings of the National Academy of Sciences.

[22]  L. Cosmides,et al.  Category-specific attention for animals reflects ancestral priorities, not expertise , 2007, Proceedings of the National Academy of Sciences.

[23]  Lars Riecke,et al.  Parietal and superior frontal visuospatial maps activated by pointing and saccades , 2007, NeuroImage.

[24]  David P Vinson,et al.  Semantic feature production norms for a large set of objects and events , 2008, Behavior research methods.

[25]  Paul T. Sowden,et al.  Your hand or mine? The extrastriate body area , 2008, NeuroImage.

[26]  Nikolaus Kriegeskorte,et al.  Frontiers in Systems Neuroscience Systems Neuroscience , 2022 .

[27]  Keiji Tanaka,et al.  Matching Categorical Object Representations in Inferior Temporal Cortex of Man and Monkey , 2008, Neuron.

[28]  W. Singer,et al.  Cortical responses to self and others , 2009, Human brain mapping.

[29]  Alison J. Wiggett,et al.  Dissociation of extrastriate body and biological-motion selective areas by manipulation of visual-motor congruency , 2009, Neuropsychologia.

[30]  G. Orban,et al.  Human Functional Magnetic Resonance Imaging Reveals Separation and Integration of Shape and Motion Cues in Biological Motion Processing , 2009, The Journal of Neuroscience.

[31]  Angela R. Laird,et al.  ALE meta-analysis of action observation and imitation in the human brain , 2010, NeuroImage.

[32]  Alison J. Wiggett,et al.  Surface-Based Information Mapping Reveals Crossmodal Vision–Action Representations in Human Parietal and Occipitotemporal Cortex , 2010, Journal of neurophysiology.

[33]  G. Orban,et al.  Coding observed motor acts: different organizational principles in the parietal and premotor cortex of humans. , 2010, Journal of neurophysiology.

[34]  Alison J. Wiggett,et al.  fMRI-adaptation studies of viewpoint tuning in the extrastriate and fusiform body areas. , 2010, Journal of neurophysiology.

[35]  Tomáš Paus,et al.  Neural Correlates of Human Body Perception , 2010, Journal of Cognitive Neuroscience.

[36]  Paul E. Downing,et al.  A comparison of volume-based and surface-based multi-voxel pattern analysis , 2011, NeuroImage.

[37]  Jens Schwarzbach,et al.  A simple framework (ASF) for behavioral and neuroimaging experiments based on the psychophysics toolbox for MATLAB , 2011, Behavior research methods.

[38]  Timothy O. Laumann,et al.  Informatics and Data Mining Tools and Strategies for the Human Connectome Project , 2011, Front. Neuroinform..

[39]  Jochen Ditterich,et al.  Splash: A Software Tool for Stereotactic Planning of Recording Chamber Placement and Electrode Trajectories , 2011, Front. Neuroinform..

[40]  J. S. Guntupalli,et al.  The Representation of Biological Classes in the Human Brain , 2012, The Journal of Neuroscience.

[41]  Paul E. Downing,et al.  Viewpoint (In)dependence of Action Representations: An MVPA Study , 2012, Journal of Cognitive Neuroscience.

[42]  Hannah E. Smithson,et al.  Modulation of the face- and body-selective visual regions by the motion and emotion of point-light face and body stimuli , 2012, NeuroImage.

[43]  H. Hampel,et al.  Neural adaptation to thin and fat bodies in the fusiform body area and middle occipital gyrus: An fMRI adaptation study , 2013, Human brain mapping.

[44]  A. Maslow A Theory of Human Motivation , 1943 .

[45]  M. Peelen,et al.  Body and Object Effectors: The Organization of Object Representations in High-Level Visual Cortex Reflects Body–Object Interactions , 2013, Journal of Neuroscience.

[46]  R. Goebel,et al.  Local Discriminability Determines the Strength of Holistic Processing for Faces in the Fusiform Face Area , 2013, Front. Psychology.

[47]  Anjan Chatterjee,et al.  Action Concepts in the Brain: An Activation Likelihood Estimation Meta-analysis , 2013, Journal of Cognitive Neuroscience.

[48]  Guy A. Orban,et al.  The overlap of the EBA and the MT/V5 cluster , 2013, NeuroImage.

[49]  R. Goebel,et al.  Human Object-Similarity Judgments Reflect and Transcend the Primate-IT Object Representation , 2013, Front. Psychol..

[50]  Mark Jenkinson,et al.  The minimal preprocessing pipelines for the Human Connectome Project , 2013, NeuroImage.

[51]  G. Orban,et al.  Common and segregated processing of observed actions in human SPL. , 2013, Cerebral cortex.

[52]  Yi Chen,et al.  Statistical inference and multiple testing correction in classification-based multi-voxel pattern analysis (MVPA): Random permutations and cluster size control , 2011, NeuroImage.

[53]  Laurel J Buxbaum,et al.  Uncovering the architecture of action semantics. , 2014, Journal of experimental psychology. Human perception and performance.

[54]  A. Lingnau,et al.  Neural correlates of grasping , 2014, Front. Hum. Neurosci..

[55]  Li Su,et al.  A Toolbox for Representational Similarity Analysis , 2014, PLoS Comput. Biol..

[56]  P. Downing,et al.  The lateral occipitotemporal cortex in action , 2015, Trends in Cognitive Sciences.

[57]  Moritz F. Wurm,et al.  Decoding Actions at Different Levels of Abstraction , 2015, The Journal of Neuroscience.

[58]  Jesper Andersson,et al.  A multi-modal parcellation of human cerebral cortex , 2016, Nature.

[59]  James V. Haxby,et al.  CoSMoMVPA: Multi-Modal Multivariate Pattern Analysis of Neuroimaging Data in Matlab/GNU Octave , 2016, bioRxiv.

[60]  H. P. Op de Beeck,et al.  Dissociations and Associations between Shape and Category Representations in the Two Visual Pathways , 2015, The Journal of Neuroscience.

[61]  Moritz F. Wurm,et al.  Decoding Concrete and Abstract Action Representations During Explicit and Implicit Conceptual Processing. , 2016, Cerebral cortex.

[62]  Moritz F. Wurm,et al.  Action Categories in Lateral Occipitotemporal Cortex Are Organized Along Sociality and Transitivity , 2017, The Journal of Neuroscience.

[63]  Russell A. Epstein,et al.  Neural Representations of Observed Actions Generalize across Static and Dynamic Visual Input , 2017, The Journal of Neuroscience.

[64]  Baolin Liu,et al.  Investigating the neural basis of basic human movement perception using multi-voxel pattern analysis , 2018, Experimental Brain Research.

[65]  Russell A. Epstein,et al.  Computational mechanisms underlying cortical responses to the affordance properties of visual scenes , 2017, bioRxiv.

[66]  Moritz F. Wurm,et al.  Distinct roles of temporal and frontoparietal cortex in representing actions across vision and language , 2019, Nature Communications.

[67]  Alfonso Caramazza,et al.  Lateral occipitotemporal cortex encodes perceptual components of social actions rather than abstract representations of sociality , 2019, NeuroImage.

[68]  Alfonso Caramazza,et al.  Directional tuning for eye and arm movements in overlapping regions in human posterior parietal cortex , 2019, NeuroImage.

[69]  A. Lingnau,et al.  The Large-Scale Organization of Gestures and Words in the Middle Temporal Gyrus , 2019, The Journal of Neuroscience.