Perceptual restoration of masked speech in human cortex

Humans are adept at understanding speech despite the fact that our natural listening environment is often filled with interference. An example of this capacity is phoneme restoration, in which part of a word is completely replaced by noise, yet listeners report hearing the whole word. The neurological basis for this unconscious fill-in phenomenon is unknown, despite being a fundamental characteristic of human hearing. Here, using direct cortical recordings in humans, we demonstrate that missing speech is restored at the acoustic-phonetic level in bilateral auditory cortex, in real-time. This restoration is preceded by specific neural activity patterns in a separate language area, left frontal cortex, which predicts the word that participants later report hearing. These results demonstrate that during speech perception, missing acoustic content is synthesized online from the integration of incoming sensory cues and the internal neural dynamics that bias word-level expectation and prediction.

[1]  E. Formisano,et al.  Hearing an Illusory Vowel in Noise: Suppression of Auditory Cortical Activity , 2012, The Journal of Neuroscience.

[2]  Josh H. McDermott,et al.  Distinct Cortical Pathways for Music and Speech Revealed by Hypothesis-Free Voxel Decomposition , 2015, Neuron.

[3]  S. Scott,et al.  Inferior Frontal Gyrus Activation Predicts Individual Differences in Perceptual Learning of Cochlear-Implant Simulations , 2010, The Journal of Neuroscience.

[4]  Josh H McDermott,et al.  Spectral completion of partially masked sounds , 2008, Proceedings of the National Academy of Sciences.

[5]  Ingrid S. Johnsrude,et al.  Illusory Vowels Resulting from Perceptual Continuity: A Functional Magnetic Resonance Imaging Study , 2008, Journal of Cognitive Neuroscience.

[6]  N. Mesgarani,et al.  Selective cortical representation of attended speaker in multi-talker speech perception , 2012, Nature.

[7]  Mitchell Steinschneider,et al.  Intracranial study of speech-elicited activity on the human posterolateral superior temporal gyrus. , 2011, Cerebral cortex.

[8]  S. David,et al.  Influence of context and behavior on stimulus reconstruction from neural activity in primary auditory cortex. , 2009, Journal of neurophysiology.

[9]  Karl J. Friston,et al.  Structural and Functional Brain Networks: From Connections to Cognition , 2013, Science.

[10]  Matthew S. Fifer,et al.  Cortical subnetwork dynamics during human language tasks , 2016, NeuroImage.

[11]  C. Gilbert,et al.  Brain States: Top-Down Influences in Sensory Processing , 2007, Neuron.

[12]  Ingrid S. Johnsrude,et al.  Human auditory cortex is sensitive to the perceived clarity of speech , 2012, NeuroImage.

[13]  D Norris,et al.  Merging information in speech recognition: Feedback is never necessary , 2000, Behavioral and Brain Sciences.

[14]  J. M. Ackroff,et al.  Auditory Induction: Perceptual Synthesis of Absent Sounds , 1972, Science.

[15]  Stephen Grossberg,et al.  Laminar cortical dynamics of conscious speech perception: neural model of phonemic restoration using subsequent context in noise. , 2011, The Journal of the Acoustical Society of America.

[16]  Karl J. Friston The free-energy principle: a unified brain theory? , 2010, Nature Reviews Neuroscience.

[17]  R. M. Warren Perceptual Restoration of Missing Speech Sounds , 1970, Science.

[18]  Matthew H. Davis,et al.  Predictive Top-Down Integration of Prior Knowledge during Speech Perception , 2012, The Journal of Neuroscience.

[19]  Antoine J. Shahin,et al.  Neural mechanisms for illusory filling-in of degraded speech , 2009, NeuroImage.

[20]  Yoshitaka Nakajima,et al.  Auditory Scene Analysis: The Perceptual Organization of Sound Albert S. Bregman , 1992 .

[21]  K. O’Connor,et al.  Illusory Sound Perception in Macaque Monkeys , 2003, The Journal of Neuroscience.

[22]  Edward F Chang,et al.  The auditory representation of speech sounds in human motor cortex , 2016, eLife.

[23]  James L. McClelland,et al.  The TRACE model of speech perception , 1986, Cognitive Psychology.

[24]  C. Eulitz,et al.  Top-down knowledge supports the retrieval of lexical information from degraded speech , 2007, Brain Research.

[25]  Keith Johnson,et al.  Phonetic Feature Encoding in Human Superior Temporal Gyrus , 2014, Science.

[26]  Andreas Kleinschmidt,et al.  Spontaneous local variations in ongoing neural activity bias perceptual decisions , 2008, Proceedings of the National Academy of Sciences.

[27]  S. Scott,et al.  Functional Integration across Brain Regions Improves Speech Perception under Adverse Listening Conditions , 2007, The Journal of Neuroscience.

[28]  Karl J. Friston,et al.  Reduced frontotemporal functional connectivity in schizophrenia associated with auditory hallucinations , 2002, Biological Psychiatry.

[29]  W. Newsome,et al.  Context-dependent computation by recurrent dynamics in prefrontal cortex , 2013, Nature.

[30]  Andreas Kleinschmidt,et al.  Functional interactions between intrinsic brain activity and behavior , 2013, NeuroImage.

[31]  Matthew K. Leonard,et al.  Dynamic Encoding of Speech Sequence Probability in Human Temporal Cortex , 2015, The Journal of Neuroscience.

[32]  J. Obleser,et al.  Expectancy constraints in degraded speech modulate the language comprehension network. , 2010, Cerebral cortex.

[33]  I. Ial,et al.  Nature Communications , 2010, Nature Cell Biology.

[34]  H. Nusbaum,et al.  Speech perception as an active cognitive process , 2014, Front. Syst. Neurosci..

[35]  Paul Boersma,et al.  Praat: doing phonetics by computer , 2003 .

[36]  P. Boersma Praat : doing phonetics by computer (version 5.1.05) , 2009 .

[37]  A. Samuel Lexical uniqueness effects on phonemic restoration , 1987 .

[38]  G. A. Miller,et al.  The Intelligibility of Interrupted Speech , 1948 .

[39]  C. Tallon-Baudry,et al.  How Ongoing Fluctuations in Human Visual Cortex Predict Perceptual Awareness: Baseline Shift versus Decision Bias , 2009, The Journal of Neuroscience.

[40]  E. Miller,et al.  An integrative theory of prefrontal cortex function. , 2001, Annual review of neuroscience.

[41]  Lori L. Holt,et al.  Speech perception under adverse conditions: insights from behavioral, computational, and neuroscience research , 2014, Front. Syst. Neurosci..

[42]  R. Murray,et al.  Increased blood flow in Broca's area during auditory hallucinations in schizophrenia , 1993, The Lancet.

[43]  Jonathan G. Fiscus,et al.  DARPA TIMIT:: acoustic-phonetic continuous speech corpus CD-ROM, NIST speech disc 1-1.1 , 1993 .

[44]  J. Fritz,et al.  Rapid task-related plasticity of spectrotemporal receptive fields in primary auditory cortex , 2003, Nature Neuroscience.

[45]  A. Samuel Phonemic restoration: insights from a new methodology. , 1981, Journal of experimental psychology. General.

[46]  Paul Boersma,et al.  Praat, a system for doing phonetics by computer , 2002 .

[47]  Simon D Lilburn,et al.  Predicting Perceptual Decision Biases from Early Brain Activity , 2012, The Journal of Neuroscience.

[48]  Byron M. Yu,et al.  Dimensionality reduction for large-scale neural recordings , 2014, Nature Neuroscience.