Neural coding of speech and language : fMRI and EEG studies

• A submitted manuscript is the version of the article upon submission and before peer-review. There can be important differences between the submitted version and the official published version of record. People interested in the research are advised to contact the author for the final version of the publication, or visit the DOI to the publisher's website. • The final author version and the galley proof are versions of the publication after peer review. • The final published version features the final layout of the paper including the volume, issue and page numbers.

[1]  James L. McClelland,et al.  The TRACE model of speech perception , 1986, Cognitive Psychology.

[2]  Rajeev D. S. Raizada,et al.  Selective Amplification of Stimulus Differences during Categorical Processing of Speech , 2007, Neuron.

[3]  René J. Huster,et al.  Methods for Simultaneous EEG-fMRI: An Introductory Review , 2012, The Journal of Neuroscience.

[4]  Adam Gazzaley,et al.  Measuring functional connectivity during distinct stages of a cognitive task , 2004, NeuroImage.

[5]  A. Postma.  Detection of errors during speech production: a review of speech monitoring models , 2000, Cognition.

[6]  M. Turvey,et al.  The motor theory of speech perception reviewed , 2006, Psychonomic bulletin & review.

[7]  Angela D. Friederici,et al.  Exploring the Activation of Semantic and Phonological Codes during Speech Planning with Event-Related Brain Potentials , 2002, Journal of Cognitive Neuroscience.

[8]  Klas Ihme,et al.  A Dry EEG-System for Scientific Research and Brain–Computer Interfaces , 2010, Front. Neurosci..

[9]  M. Kutas,et al.  Electrophysiological estimates of the time course of semantic and phonological encoding during implicit picture naming. , 2000, Psychophysiology.

[10]  Richard J. Binney,et al.  The ventral and inferolateral aspects of the anterior temporal lobe are crucial in semantic memory: evidence from a novel direct comparison of distortion-corrected fMRI, rTMS, and semantic dementia. , 2010, Cerebral cortex.

[11]  Kayoko Okada,et al.  Area Spt in the Human Planum Temporale Supports Sensory-motor Integration for Speech Processing Establishing the Existence of Distinct Sen- Sory versus Motor Activation Patterns Would Establish That , 2022 .

[12]  Rainer Goebel,et al.  "Who" Is Saying "What"? Brain-Based Decoding of Human Voice and Speech , 2008, Science.

[13]  M. Torrens.  Co-Planar Stereotaxic Atlas of the Human Brain—3-Dimensional Proportional System: An Approach to Cerebral Imaging, J. Talairach, P. Tournoux. Georg Thieme Verlag, New York (1988), 122 pp., 130 figs. DM 268 , 1990 .

[14]  Eric Halgren,et al.  First-Pass Selectivity for Semantic Categories in Human Anteroventral Temporal Lobe , 2011, The Journal of Neuroscience.

[15]  Rainer Goebel,et al.  Analysis of functional image analysis contest (FIAC) data with brainvoyager QX: From single‐subject to cortically aligned group general linear model analysis and self‐organizing group independent component analysis , 2006, Human brain mapping.

[16]  M. Kutas,et al.  Reading senseless sentences: brain potentials reflect semantic incongruity. , 1980, Science.

[17]  G. Rizzolatti,et al.  Speech listening specifically modulates the excitability of tongue muscles: a TMS study , 2002, The European journal of neuroscience.

[18]  Giancarlo Valente,et al.  Task-Dependent Decoding of Speaker and Vowel Identity from Auditory Cortical Response Patterns , 2014, The Journal of Neuroscience.

[19]  A. Friederici,et al.  Auditory perception and syntactic cognition: brain activity‐based decoding within and across subjects , 2012, The European journal of neuroscience.

[20]  A. Baddeley,et al.  The phonological loop as a language learning device. , 1998, Psychological review.

[21]  R. Salmelin,et al.  Time course of top-down and bottom-up influences on syllable processing in the auditory cortex. , 2006, Cerebral cortex.

[22]  Sophie K. Scott,et al.  The functional neuroanatomy of prelexical processing in speech perception , 2004, Cognition.

[23]  Lloyd T. Elliott,et al.  Cortical surface-based searchlight decoding , 2011, NeuroImage.

[24]  R. Zatorre,et al.  Voice-selective areas in human auditory cortex , 2000, Nature.

[25]  Kara D. Federmeier,et al.  Electrophysiology reveals semantic memory use in language comprehension , 2000, Trends in Cognitive Sciences.

[26]  Frank H. Guenther,et al.  A neural theory of speech acquisition and production , 2012, Journal of Neurolinguistics.

[27]  W. Levelt,et al.  The spatial and temporal signatures of word production components , 2004, Cognition.

[28]  Stuart Rosen,et al.  A positron emission tomography study of the neural basis of informational and energetic masking effects in speech perception. , 2004, The Journal of the Acoustical Society of America.

[29]  Giancarlo Valente,et al.  Dynamic and Task-Dependent Encoding of Speech and Voice by Phase Reorganization of Cortical Oscillations , 2009, The Journal of Neuroscience.

[30]  Karl J. Friston,et al.  Psychophysiological and Modulatory Interactions in Neuroimaging , 1997, NeuroImage.

[31]  Harold Goodglass,et al.  Interaction between phonological and semantic factors in auditory comprehension , 1981, Neuropsychologia.

[32]  R. Turner,et al.  Language Control in the Bilingual Brain , 2006, Science.

[33]  Rainer Goebel,et al.  Combining multivariate voxel selection and support vector machines for mapping and classification of fMRI spatial patterns , 2008, NeuroImage.

[34]  Elia Formisano,et al.  The Sensory Consequences of Speaking: Parametric Neural Cancellation during Speech in Auditory Cortex , 2011, PloS one.

[35]  E. Formisano,et al.  Learning of New Sound Categories Shapes Neural Response Patterns in Human Auditory Cortex , 2012, The Journal of Neuroscience.

[36]  Bryan R. Conroy,et al.  A Common, High-Dimensional Model of the Representational Space in Human Ventral Temporal Cortex , 2011, Neuron.

[37]  M. Petrides,et al.  Cortico-cortical connections of areas 44 and 45B in the macaque monkey , 2014, Brain and Language.

[38]  A. Baddeley.  Working memory and language: an overview. , 2003, Journal of communication disorders.

[39]  S. Saxena,et al.  Psychologists’ perspectives on the diagnostic classification of mental disorders: Results from the WHO-IUPsyS Global Survey , 2013, International journal of psychology : Journal international de psychologie.

[40]  Rainer Goebel,et al.  Measuring structural–functional correspondence: Spatial variability of specialised brain regions after macro-anatomical alignment , 2012, NeuroImage.

[41]  M. Maj,et al.  Comparison of Chinese and international psychiatrists' views on classification of mental disorders , 2014, Asia-Pacific psychiatry : official journal of the Pacific Rim College of Psychiatrists.

[42]  A. Caramazza,et al.  The organisation of conceptual knowledge in the brain: The future's past and some future directions , 2006, Cognitive neuropsychology.

[43]  Lourens J. Waldorp,et al.  Effective connectivity of fMRI data using ancestral graph theory: Dealing with missing regions , 2011, NeuroImage.

[44]  Matthew F Glasser,et al.  DTI tractography of the human brain's language pathways. , 2008, Cerebral cortex.

[45]  P. Hagoort.  The fractionation of spoken language understanding by measuring electrical and magnetic brain signals , 2008, Philosophical Transactions of the Royal Society B: Biological Sciences.

[46]  R. Oostenveld,et al.  I see what you mean: Theta power increases are involved in the retrieval of lexical semantic information , 2008, Brain and Language.

[47]  Gregory Hickok,et al.  The role of mirror neurons in speech and language processing , 2010, Brain and Language.

[48]  Sophie K. Scott,et al.  What is the relationship between phonological short-term memory and speech processing? , 2006, Trends in Cognitive Sciences.

[49]  T. Sejnowski,et al.  Dynamic Brain Sources of Visual Evoked Responses , 2002, Science.

[50]  Cathy J. Price,et al.  Sensory-to-motor integration during auditory repetition: a combined fMRI and lesion study , 2014, Front. Hum. Neurosci..

[51]  W. Marslen-Wilson.  Functional parallelism in spoken word-recognition , 1987, Cognition.

[52]  David Poeppel,et al.  Acoustic landmarks drive delta–theta oscillations to enable speech comprehension by facilitating perceptual parsing , 2014, NeuroImage.

[53]  Kayoko Okada,et al.  Conduction aphasia, sensory-motor integration, and phonological short-term memory – An aggregate analysis of lesion and fMRI data , 2011, Brain and Language.

[54]  Lee M. Miller,et al.  Measuring interregional functional connectivity using coherence and partial coherence analyses of fMRI data , 2004, NeuroImage.

[55]  Elizabeth Jefferies,et al.  The Selective Role of Premotor Cortex in Speech Perception: A Contribution to Phoneme Judgements but not Speech Comprehension , 2013, Journal of Cognitive Neuroscience.

[56]  Ankoor S. Shah,et al.  An oscillatory hierarchy controlling neuronal excitability and stimulus processing in the auditory cortex. , 2005, Journal of neurophysiology.

[57]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[58]  Karl J. Friston.  Functional and Effective Connectivity: A Review , 2011, Brain Connect..

[59]  A. Liberman,et al.  The motor theory of speech perception revised , 1985, Cognition.

[60]  Lars Hausfeld,et al.  EEG decoding of spoken words in bilingual listeners: from words to language invariant semantic-conceptual representations , 2015, Front. Psychol..

[61]  Marc Brysbaert,et al.  A validation study of the age-of-acquisition norms collected by Ghyselinck, De Moor, & Brysbaert , 2000 .

[62]  A. Damasio.  Time-locked multiregional retroactivation: A systems-level proposal for the neural substrates of recall and recognition , 1989, Cognition.

[63]  R. H. Baayen,et al.  The CELEX Lexical Database (CD-ROM) , 1996 .

[64]  A. Damasio,et al.  A neural basis for lexical retrieval , 1996, Nature.

[65]  Tom M. Mitchell,et al.  Machine learning classifiers and fMRI: A tutorial overview , 2009, NeuroImage.

[66]  Robert Oostenveld,et al.  Modality-independent decoding of semantic information from the human brain. , 2014, Cerebral cortex.

[67]  Sharon L. Thompson-Schill,et al.  Informational connectivity: identifying synchronized discriminability of multi-voxel patterns across the brain , 2013, Front. Hum. Neurosci..

[68]  J. Obleser,et al.  Frequency modulation entrains slow neural oscillations and optimizes human listening behavior , 2012, Proceedings of the National Academy of Sciences.

[69]  Karl J. Friston,et al.  Dynamic causal modelling , 2003, NeuroImage.

[70]  Terrence J. Sejnowski,et al.  Independent Component Analysis Using an Extended Infomax Algorithm for Mixed Subgaussian and Supergaussian Sources , 1999, Neural Computation.

[71]  R. Goebel,et al.  Individual faces elicit distinct response patterns in human anterior temporal cortex , 2007, Proceedings of the National Academy of Sciences.

[72]  David D. Cox,et al.  Functional magnetic resonance imaging (fMRI) “brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual cortex , 2003, NeuroImage.

[73]  Maxime Descoteaux,et al.  Regional variations in vascular density correlate with resting‐state and task‐evoked blood oxygen level‐dependent signal amplitude , 2014, Human brain mapping.

[74]  I. Christoffels,et al.  Language conflict in translation: An ERP study of translation production , 2013 .

[75]  Valerie A. Carr,et al.  Spatiotemporal Dynamics of Modality-Specific and Supramodal Word Processing , 2003, Neuron.

[76]  Michael Eichler,et al.  A graphical approach for evaluating effective connectivity in neural systems , 2005, Philosophical Transactions of the Royal Society B: Biological Sciences.

[77]  Elizabeth Jefferies,et al.  Both the Middle Temporal Gyrus and the Ventral Anterior Temporal Area Are Crucial for Multimodal Semantic Processing: Distortion-corrected fMRI Evidence for a Double Gradient of Information Convergence in the Temporal Lobes , 2012, Journal of Cognitive Neuroscience.

[78]  Yinjuan Du,et al.  Noise differentially impacts phoneme representations in the auditory and speech motor systems , 2014, Proceedings of the National Academy of Sciences.

[79]  José Miguel Salles Dias,et al.  Semi-Automatic 3D Reconstruction of Urban Areas Using Epipolar Geometry and Template Matching , 2006, Comput. Aided Civ. Infrastructure Eng..

[80]  R. Salmelin.  Clinical neurophysiology of language: The MEG approach , 2007, Clinical Neurophysiology.

[81]  Chris Rorden,et al.  Temporal Order Processing of Syllables in the Left Parietal Lobe , 2009, The Journal of Neuroscience.

[82]  B. Mesquita,et al.  Adjustment to Chronic Diseases and Terminal Illness Health Psychology : Psychological Adjustment to Chronic Disease , 2006 .

[83]  J. Ayuso-Mateos,et al.  Mental health professionals' natural taxonomies of mental disorders: implications for the clinical utility of the ICD-11 and the DSM-5. , 2013, Journal of clinical psychology.

[84]  Tom M. Mitchell,et al.  From the SelectedWorks of Marcel Adam Just 2011 Commonality of neural representations of words and pictures , 2016 .

[85]  A. Ishai,et al.  Distributed and Overlapping Representations of Faces and Objects in Ventral Temporal Cortex , 2001, Science.

[86]  Arnaud Delorme,et al.  EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis , 2004, Journal of Neuroscience Methods.

[87]  S. Saxena,et al.  The WPA‐WHO Global Survey of Psychiatrists’ Attitudes Towards Mental Disorders Classification , 2011, World psychiatry : official journal of the World Psychiatric Association.

[88]  Friedemann Pulvermüller,et al.  Causal Influence of Articulatory Motor Cortex on Comprehending Single Spoken Words: TMS Evidence , 2014, Cerebral cortex.

[89]  Keith Johnson,et al.  Phonetic Feature Encoding in Human Superior Temporal Gyrus , 2014, Science.

[90]  Rainer Goebel,et al.  Predicting EEG single trial responses with simultaneous fMRI and Relevance Vector Machine regression , 2011, NeuroImage.

[91]  矫立志.  Language learning device , 2013 .

[92]  Marco Iacoboni,et al.  The Essential Role of Premotor Cortex in Speech Perception , 2007, Current Biology.

[93]  E. T. Possing,et al.  Human temporal lobe activation by speech and nonspeech sounds. , 2000, Cerebral cortex.

[94]  Tom M. Mitchell,et al.  Identifying bilingual semantic neural representations across languages , 2012, Brain and Language.

[95]  Paul E. Downing,et al.  A comparison of volume-based and surface-based multi-voxel pattern analysis , 2011, NeuroImage.

[96]  James M. McQueen,et al.  Decoding Speech Perception by Native and Non-Native Speakers Using Single-Trial Electrophysiological Data , 2013, PloS one.

[97]  David Poeppel,et al.  Cortical oscillations and speech processing: emerging computational principles and operations , 2012, Nature Neuroscience.

[98]  Dirk Koester,et al.  Morphological priming in overt language production: Electrophysiological evidence from Dutch , 2008, NeuroImage.

[99]  Kristofer E. Bouchard,et al.  Functional Organization of Human Sensorimotor Cortex for Speech Articulation , 2013, Nature.

[100]  Peter Indefrey,et al.  The use of conceptual components in language production: an ERP study , 2014, Front. Psychol..

[101]  Rainer Goebel,et al.  Information-based functional brain mapping. , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[102]  J. Rauschecker,et al.  Maps and streams in the auditory cortex: nonhuman primates illuminate human speech processing , 2009, Nature Neuroscience.

[103]  Rainer Goebel,et al.  Mapping directed influence over the brain using Granger causality and fMRI , 2005, NeuroImage.

[104]  Giancarlo Valente,et al.  Brain-Based Translation: fMRI Decoding of Spoken Words in Bilinguals Reveals Language-Independent Semantic Representations in Anterior Temporal Lobe , 2014, The Journal of Neuroscience.

[105]  T. Rogers,et al.  Where do you know what you know? The representation of semantic knowledge in the human brain , 2007, Nature Reviews Neuroscience.

[106]  David Friedman,et al.  Single-trial discrimination for integrating simultaneous EEG and fMRI: Identifying cortical areas contributing to trial-to-trial variability in the auditory oddball task , 2009, NeuroImage.

[107]  Sonja A. Kotz,et al.  Alpha and theta brain oscillations index dissociable processes in spoken word recognition , 2014, NeuroImage.

[108]  Hanna Damasio,et al.  Predicting visual stimuli on the basis of activity in auditory cortices , 2010, Nature Neuroscience.

[109]  M. Iacoboni,et al.  Listening to speech activates motor areas involved in speech production , 2004, Nature Neuroscience.

[110]  Feng Rong,et al.  Sensorimotor Integration in Speech Processing: Computational Basis and Neural Organization , 2011, Neuron.

[111]  Vladimir I. Levenshtein,et al.  Binary codes capable of correcting deletions, insertions, and reversals , 1965 .

[112]  F. Aboitiz.  Gestures, Vocalizations, and Memory in Language Origins , 2011, Front. Evol. Neurosci..

[113]  A. Damasio,et al.  Convergence and divergence in a neural architecture for recognition and memory , 2009, Trends in Neurosciences.

[114]  Tom Michael Mitchell,et al.  Predicting Human Brain Activity Associated with the Meanings of Nouns , 2008, Science.

[115]  Giancarlo Valente,et al.  Multivariate analysis of fMRI time series: classification and regression of brain responses using machine learning. , 2008, Magnetic resonance imaging.

[116]  Rasha Abdel Rahman,et al.  Electrophysiological Chronometry of Semantic Context Effects in Language Production , 2011, Journal of Cognitive Neuroscience.

[117]  Paul Boersma,et al.  Praat: doing phonetics by computer , 2003 .

[118]  Kristin Lemhöfer,et al.  Introducing LexTALE: A quick and valid Lexical Test for Advanced Learners of English , 2011, Behavior research methods.

[119]  L. Fadiga,et al.  The Motor Somatotopy of Speech Perception , 2009, Current Biology.

[120]  Srikantan S. Nagarajan,et al.  Speech Production as State Feedback Control , 2011, Front. Hum. Neurosci..

[121]  G. Hickok,et al.  AuditoryMotor Interaction Revealed by fMRI: Speech, Music, and Working Memory in Area Spt , 2003, Journal of Cognitive Neuroscience.

[122]  Milene Bonte,et al.  Developmental changes in ERP correlates of spoken word recognition during early school years: a phonological priming study , 2004, Clinical Neurophysiology.

[123]  W. Levelt,et al.  Speaking: From Intention to Articulation , 1990 .

[124]  L. Fadiga,et al.  Active perception: sensorimotor circuits as a cortical basis for language , 2010, Nature Reviews Neuroscience.

[125]  G. Waters,et al.  On the Nature of the Phonological Output Planning Processes Involved in Verbal Rehearsal: Evidence from Aphasia , 1995, Brain and Language.

[126]  Lars Hausfeld,et al.  Pattern analysis of EEG responses to speech and voice: Influence of feature grouping , 2012, NeuroImage.

[127]  Magdalena Szumilas.  Explaining odds ratios. , 2010, Journal of the Canadian Academy of Child and Adolescent Psychiatry = Journal de l'Academie canadienne de psychiatrie de l'enfant et de l'adolescent.

[128]  Jonas Obleser,et al.  Attentional influences on functional mapping of speech sounds in human auditory cortex , 2004, BMC Neuroscience.

[129]  Markus Butz,et al.  Task-dependent oscillations during unimanual and bimanual movements in the human primary motor cortex and SMA studied with magnetoencephalography , 2005, NeuroImage.

[130]  Lucia Melloni,et al.  Brain Oscillations during Spoken Sentence Processing , 2012, Journal of Cognitive Neuroscience.

[131]  Ayse Pinar Saygin,et al.  Smoothing and cluster thresholding for cortical surface-based group analysis of fMRI data , 2006, NeuroImage.

[132]  Alan C. Evans,et al.  Lateralization of phonetic and pitch discrimination in speech processing. , 1992, Science.

[133]  Matthew A. Lambon Ralph,et al.  Differential Contributions of Bilateral Ventral Anterior Temporal Lobe and Left Anterior Superior Temporal Gyrus to Semantic Processes , 2011, Journal of Cognitive Neuroscience.

[134]  Colin M. Brown,et al.  Electrophysiological Evidence for Early Contextual Influences during Spoken-Word Recognition: N200 Versus N400 Effects , 2001, Journal of Cognitive Neuroscience.

[135]  D. Poeppel,et al.  The cortical organization of speech processing , 2007, Nature Reviews Neuroscience.

[136]  Jessica S. Arsenault,et al.  Distributed Neural Representations of Phonological Features during Speech Perception , 2015, The Journal of Neuroscience.

[137]  Lisa D. Sanders,et al.  An ERP study of continuous speech processing. I. Segmentation, semantics, and syntax in native speakers. , 2003, Brain research. Cognitive brain research.

[138]  Sydney S. Cash,et al.  Decoding word and category-specific spatiotemporal representations from MEG and EEG , 2011, NeuroImage.

[139]  Jonathan D. Cohen,et al.  Improved Assessment of Significant Activation in Functional Magnetic Resonance Imaging (fMRI): Use of a Cluster‐Size Threshold , 1995, Magnetic resonance in medicine.

[140]  D. Poeppel,et al.  Phase Patterns of Neuronal Responses Reliably Discriminate Speech in Human Auditory Cortex , 2007, Neuron.

[141]  Robert T. Knight,et al.  A method for event-related phase/amplitude coupling , 2013, NeuroImage.

[142]  P. Hagoort.  On Broca, brain, and binding: a new framework , 2005, Trends in Cognitive Sciences.

[143]  E. Formisano,et al.  Auditory Cortex Encodes the Perceptual Interpretation of Ambiguous Sound , 2011, The Journal of Neuroscience.

[144]  Alard Roebroeck,et al.  General overview on the merits of multimodal neuroimaging data fusion , 2014, NeuroImage.

[145]  C. Price.  The anatomy of language: a review of 100 fMRI studies published in 2009 , 2010, Annals of the New York Academy of Sciences.

[146]  William W. Graves,et al.  Where is the semantic system? A critical review and meta-analysis of 120 functional neuroimaging studies. , 2009, Cerebral cortex.

[147]  Matthew H. Davis,et al.  Neural Oscillations Carry Speech Rhythm through to Comprehension , 2012, Front. Psychology.

[148]  Friedemann Pulvermüller,et al.  Motor cortex maps articulatory features of speech sounds , 2006, Proceedings of the National Academy of Sciences of the United States of America.