Decoding Articulatory Features from fMRI Responses in Dorsal Speech Regions

The brain's circuitry for perceiving and producing speech may show a notable level of overlap that is crucial for normal development and behavior. The extent to which sensorimotor integration plays a role in speech perception remains highly controversial, however. Methodological constraints related to experimental designs and analysis methods have so far prevented the disentanglement of neural responses to acoustic versus articulatory speech features. Using a passive listening paradigm and multivariate decoding of single-trial fMRI responses to spoken syllables, we investigated brain-based generalization of articulatory features (place and manner of articulation, and voicing) beyond their acoustic (surface) form in adult human listeners. For example, we trained a classifier to discriminate place of articulation within stop syllables (e.g., /pa/ vs /ta/) and tested whether this training generalizes to fricatives (e.g., /fa/ vs /sa/). This novel approach revealed generalization of place and manner of articulation at multiple cortical levels within the dorsal auditory pathway, including auditory, sensorimotor, motor, and somatosensory regions, suggesting the representation of sensorimotor information. Additionally, generalization of voicing included the right anterior superior temporal sulcus associated with the perception of human voices as well as somatosensory regions bilaterally. Our findings highlight the close connection between brain systems for speech perception and production, and in particular, indicate the availability of articulatory codes during passive speech perception. SIGNIFICANCE STATEMENT Sensorimotor integration is central to verbal communication and provides a link between auditory signals of speech perception and motor programs of speech production. It remains highly controversial, however, to what extent the brain's speech perception system actively uses articulatory (motor), in addition to acoustic/phonetic, representations. In this study, we examine the role of articulatory representations during passive listening using carefully controlled stimuli (spoken syllables) in combination with multivariate fMRI decoding. Our approach enabled us to disentangle brain responses to acoustic and articulatory speech properties. In particular, it revealed articulatory-specific brain responses of speech at multiple cortical levels, including auditory, sensorimotor, and motor regions, suggesting the representation of sensorimotor information during passive speech perception.

[1]  Feng Rong,et al.  Sensorimotor Integration in Speech Processing: Computational Basis and Neural Organization , 2011, Neuron.

[2]  D. N. Pandya,et al.  Further observations on parieto-temporal connections in the rhesus monkey , 2004, Experimental Brain Research.

[3]  Rainer Goebel,et al.  "Who" Is Saying "What"? Brain-Based Decoding of Human Voice and Speech , 2008, Science.

[4]  Matthew H. Davis,et al.  Brain regions recruited for the effortful comprehension of noise-vocoded words , 2012 .

[5]  R. Zatorre,et al.  Voice-selective areas in human auditory cortex , 2000, Nature.

[6]  E. Formisano,et al.  Auditory Cortex Encodes the Perceptual Interpretation of Ambiguous Sound , 2011, The Journal of Neuroscience.

[7]  G. Hickok,et al.  Auditory–Motor Interaction Revealed by fMRI: Speech, Music, and Working Memory in Area Spt , 2003 .

[8]  P. Ladefoged A course in phonetics , 1975 .

[9]  Paul Boersma,et al.  Praat, a system for doing phonetics by computer , 2002 .

[10]  Rainer Goebel,et al.  Measuring structural–functional correspondence: Spatial variability of specialised brain regions after macro-anatomical alignment , 2012, NeuroImage.

[11]  C. Westin,et al.  Human middle longitudinal fascicle: variations in patterns of anatomical connections , 2013, Brain Structure and Function.

[12]  Alan C. Evans,et al.  Lateralization of phonetic and pitch discrimination in speech processing. , 1992, Science.

[13]  E. Formisano,et al.  Learning of New Sound Categories Shapes Neural Response Patterns in Human Auditory Cortex , 2012, The Journal of Neuroscience.

[14]  Marco Iacoboni,et al.  The Essential Role of Premotor Cortex in Speech Perception , 2007, Current Biology.

[15]  Yinjuan Du,et al.  Noise differentially impacts phoneme representations in the auditory and speech motor systems , 2014, Proceedings of the National Academy of Sciences.

[16]  Kristofer E. Bouchard,et al.  Functional Organization of Human Sensorimotor Cortex for Speech Articulation , 2013, Nature.

[17]  Ayse Pinar Saygin,et al.  Smoothing and cluster thresholding for cortical surface-based group analysis of fMRI data , 2006, NeuroImage.

[18]  Gregory Hickok,et al.  Eight Problems for the Mirror Neuron Theory of Action Understanding in Monkeys and Humans , 2009, Journal of Cognitive Neuroscience.

[19]  M. Torrens Co-Planar Stereotaxic Atlas of the Human Brain—3-Dimensional Proportional System: An Approach to Cerebral Imaging, J. Talairach, P. Tournoux. Georg Thieme Verlag, New York (1988), 122 pp., 130 figs. DM 268 , 1990 .

[20]  Bijan Pesaran,et al.  Sensory-motor transformations for speech occur bilaterally , 2014, Nature.

[21]  A. Baddeley,et al.  The phonological loop as a language learning device. , 1998, Psychological review.

[22]  Edward F Chang,et al.  Control of Spoken Vowel Acoustics and the Influence of Phonetic Context in Human Speech Sensorimotor Cortex , 2014, The Journal of Neuroscience.

[23]  L. Fadiga,et al.  Active perception: sensorimotor circuits as a cortical basis for language , 2010, Nature Reviews Neuroscience.

[24]  Sean M. Polyn,et al.  Beyond mind-reading: multi-voxel pattern analysis of fMRI data , 2006, Trends in Cognitive Sciences.

[25]  G. Hickok,et al.  AuditoryMotor Interaction Revealed by fMRI: Speech, Music, and Working Memory in Area Spt , 2003, Journal of Cognitive Neuroscience.

[26]  Friedemann Pulvermüller,et al.  Motor cortex maps articulatory features of speech sounds , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[27]  Matthew F Glasser,et al.  DTI tractography of the human brain's language pathways. , 2008, Cerebral cortex.

[28]  Chris Rorden,et al.  Temporal Order Processing of Syllables in the Left Parietal Lobe , 2009, The Journal of Neuroscience.

[29]  Ana B. Chica,et al.  Attentional Routes to Conscious Perception , 2012, Front. Psychology.

[30]  M. Turvey,et al.  The motor theory of speech perception reviewed , 2006, Psychonomic bulletin & review.

[31]  Richard Granger,et al.  Categorical Speech Processing in Broca's Area: An fMRI Study Using Multivariate Pattern-Based Analysis , 2012, The Journal of Neuroscience.

[32]  Kayoko Okada,et al.  Area Spt in the Human Planum Temporale Supports Sensory-motor Integration for Speech Processing Establishing the Existence of Distinct Sen- Sory versus Motor Activation Patterns Would Establish That , 2022 .

[33]  Giancarlo Valente,et al.  Brain-Based Translation: fMRI Decoding of Spoken Words in Bilinguals Reveals Language-Independent Semantic Representations in Anterior Temporal Lobe , 2014, The Journal of Neuroscience.

[34]  Kayoko Okada,et al.  Conduction aphasia, sensory-motor integration, and phonological short-term memory – An aggregate analysis of lesion and fMRI data , 2011, Brain and Language.

[35]  M. Iacoboni,et al.  Listening to speech activates motor areas involved in speech production , 2004, Nature Neuroscience.

[36]  D. Poeppel,et al.  The cortical organization of speech processing , 2007, Nature Reviews Neuroscience.

[37]  Srikantan S. Nagarajan,et al.  Speech Production as State Feedback Control , 2011, Front. Hum. Neurosci..

[38]  Sophie K. Scott,et al.  What is the relationship between phonological short-term memory and speech processing? , 2006, Trends in Cognitive Sciences.

[39]  Lars Hausfeld,et al.  EEG decoding of spoken words in bilingual listeners: from words to language invariant semantic-conceptual representations , 2015, Front. Psychol..

[40]  Tom M. Mitchell,et al.  Identifying bilingual semantic neural representations across languages , 2012, Brain and Language.

[41]  Rainer Goebel,et al.  Combining multivariate voxel selection and support vector machines for mapping and classification of fMRI spatial patterns , 2008, NeuroImage.

[42]  Rainer Goebel,et al.  Information-based functional brain mapping. , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[43]  Paul Boersma,et al.  Praat: doing phonetics by computer , 2003 .

[44]  Rainer Goebel,et al.  Analysis of functional image analysis contest (FIAC) data with brainvoyager QX: From single‐subject to cortically aligned group general linear model analysis and self‐organizing group independent component analysis , 2006, Human brain mapping.

[45]  Matthew H. Davis,et al.  Hierarchical Organization of Auditory and Motor Representations in Speech Perception: Evidence from Searchlight Similarity Analysis , 2015, Cerebral cortex.

[46]  Frank H. Guenther,et al.  A neural theory of speech acquisition and production , 2012, Journal of Neurolinguistics.

[47]  Jonathan D. Cohen,et al.  Improved Assessment of Significant Activation in Functional Magnetic Resonance Imaging (fMRI): Use of a Cluster‐Size Threshold , 1995, Magnetic resonance in medicine.

[48]  Friedemann Pulvermüller,et al.  Causal Influence of Articulatory Motor Cortex on Comprehending Single Spoken Words: TMS Evidence , 2014, Cerebral cortex.

[49]  Rajeev D. S. Raizada,et al.  Selective Amplification of Stimulus Differences during Categorical Processing of Speech , 2007, Neuron.

[50]  Tom M. Mitchell,et al.  From the SelectedWorks of Marcel Adam Just 2011 Commonality of neural representations of words and pictures , 2016 .

[51]  L. Fadiga,et al.  The Motor Somatotopy of Speech Perception , 2009, Current Biology.

[52]  Jessica S. Arsenault,et al.  Distributed Neural Representations of Phonological Features during Speech Perception , 2015, The Journal of Neuroscience.

[53]  G. Waters,et al.  On the Nature of the Phonological Output Planning Processes Involved in Verbal Rehearsal: Evidence from Aphasia , 1995, Brain and Language.

[54]  Robert Oostenveld,et al.  Modality-independent decoding of semantic information from the human brain. , 2014, Cerebral cortex.

[55]  Elizabeth Jefferies,et al.  The Selective Role of Premotor Cortex in Speech Perception: A Contribution to Phoneme Judgements but not Speech Comprehension , 2013, Journal of Cognitive Neuroscience.

[56]  N. Logothetis,et al.  A voice region in the monkey brain , 2008, Nature Neuroscience.

[57]  A. Liberman,et al.  The motor theory of speech perception revised , 1985, Cognition.

[58]  Marlene Behrmann,et al.  Unraveling the distributed neural code of facial identity through spatiotemporal pattern analysis , 2011, Proceedings of the National Academy of Sciences.

[59]  Jeffrey M. Zacks,et al.  Searchlight analysis: Promise, pitfalls, and potential , 2013, NeuroImage.

[60]  Lloyd T. Elliott,et al.  Cortical surface-based searchlight decoding , 2011, NeuroImage.

[61]  Giancarlo Valente,et al.  Task-Dependent Decoding of Speaker and Vowel Identity from Auditory Cortical Response Patterns , 2014, The Journal of Neuroscience.

[62]  Keith Johnson,et al.  Phonetic Feature Encoding in Human Superior Temporal Gyrus , 2014, Science.

[63]  M. Bangert,et al.  Perception of Words and Pitch Patterns in Song and Speech , 2012, Front. Psychology.

[64]  A. Woods,et al.  Context Modulates the Contribution of Time and Space in Causal Inference , 2012, Front. Psychology.

[65]  Paul E. Downing,et al.  A comparison of volume-based and surface-based multi-voxel pattern analysis , 2011, NeuroImage.