The neural oscillatory markers of phonetic convergence during verbal interaction

During a conversation, the neural processes supporting speech production and perception overlap in time and, based on context, expectations and the dynamics of interaction, they are also continuously modulated in real time. Recently, the growing interest in the neural dynamics underlying interactive tasks, in particular in the language domain, has mainly tackled the temporal aspects of turn‐taking in dialogs. Besides temporal coordination, an under‐investigated phenomenon is the implicit convergence of the speakers toward a shared phonetic space. Here, we used dual electroencephalography (dual‐EEG) to record brain signals from subjects involved in a relatively constrained interactive task where they were asked to take turns in chaining words according to a phonetic rhyming rule. We quantified participants' initial phonetic fingerprints and tracked their phonetic convergence during the interaction via a robust and automatic speaker verification technique. Results show that phonetic convergence is associated to left frontal alpha/low‐beta desynchronization during speech preparation and by high‐beta suppression before and during listening to speech in right centro‐parietal and left frontal sectors, respectively. By this work, we provide evidence that mutual adaptation of speech phonetic targets, correlates with specific alpha and beta oscillatory dynamics. Alpha and beta oscillatory dynamics may index the coordination of the “when” as well as the “how” speech interaction takes place, reinforcing the suggestion that perception and production processes are highly interdependent and co‐constructed during a conversation.

[1]  G. Pfurtscheller,et al.  Visualization of sensorimotor areas involved in preparation for hand movement based on classification of μ and central β rhythms in single EEG trials in man , 1994, Neuroscience Letters.

[2]  Marco Iacoboni,et al.  The Essential Role of Premotor Cortex in Speech Perception , 2007, Current Biology.

[3]  R. Krauss,et al.  Dominance and accommodation in the conversational behaviours of same- and mixed-gender dyads. , 1988 .

[4]  Cathy J. Price,et al.  A review and synthesis of the first 20 years of PET and fMRI studies of heard speech, spoken language and reading , 2012, NeuroImage.

[5]  L. Zago,et al.  Right hemisphere dominance for auditory attention and its modulation by eye position: an event related fMRI study. , 2007, Restorative neurology and neuroscience.

[6]  L. Fadiga,et al.  Automatic versus Voluntary Motor Imitation: Effect of Visual Context and Stimulus Velocity , 2010, PloS one.

[7]  Arnaud Delorme,et al.  EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis , 2004, Journal of Neuroscience Methods.

[8]  Deirdre Wilson,et al.  The mapping between the mental and the public Lexicon , 1998 .

[9]  A. D’Ausilio,et al.  Beta rhythm modulation by speech sounds: somatotopic mapping in somatosensory cortex , 2016, Scientific Reports.

[10]  R. Hari,et al.  Functional Segregation of Movement-Related Rhythmic Activity in the Human Brain , 1995, NeuroImage.

[11]  Luciano Fadiga,et al.  The Relationship Between F0 Synchrony and Speech Convergence in Dyadic Interaction , 2017, INTERSPEECH.

[12]  G. Carmignoto,et al.  Muscle reinnervation—I. Restoration of transmitter release mechanisms , 1983, Neuroscience.

[13]  R. Lesser,et al.  Functional mapping of human sensorimotor cortex with electrocorticographic spectral analysis. I. Alpha and beta event-related desynchronization. , 1998, Brain : a journal of neurology.

[14]  Maria Uther,et al.  Do you speak E-NG-L-I-SH? A comparison of foreigner- and infant-directed speech , 2007, Speech Commun..

[15]  S. Goldinger Echoes of echoes? An episodic theory of lexical access. , 1998, Psychological review.

[16]  Karl J. Friston,et al.  Action perception as hypothesis testing , 2017, Cortex.

[17]  I. Toni,et al.  Oscillations , 2018, Physics to a Degree.

[18]  Luc H. Arnal,et al.  Transitions in neural oscillations reflect prediction errors generated in audiovisual speech , 2011, Nature Neuroscience.

[19]  Sylvain Baillet,et al.  Motor origin of temporal predictions in auditory attention , 2017, Proceedings of the National Academy of Sciences.

[20]  S. Levinson Turn-taking in Human Communication – Origins and Implications for Language Processing , 2016, Trends in Cognitive Sciences.

[21]  Lauri Parkkonen,et al.  Sensorimotor activation related to speaker vs. listener role during natural conversation , 2016, Neuroscience Letters.

[22]  Sridha Sridharan,et al.  Feature warping for robust speaker verification , 2001, Odyssey.

[23]  G. Ding,et al.  Leader emergence through interpersonal neural synchronization , 2015, Proceedings of the National Academy of Sciences.

[24]  G. Pfurtscheller,et al.  Patterns of cortical activation during planning of voluntary movement. , 1989, Electroencephalography and clinical neurophysiology.

[25]  M. Pickering,et al.  Toward a mechanistic psychology of dialogue , 2004, Behavioral and Brain Sciences.

[26]  R. Oostenveld,et al.  Nonparametric statistical testing of EEG- and MEG-data , 2007, Journal of Neuroscience Methods.

[27]  U. Hasson,et al.  On the Same Wavelength: Predictable Language Enhances Speaker–Listener Brain-to-Brain Synchrony in Posterior Superior Temporal Gyrus , 2014, The Journal of Neuroscience.

[28]  Laurent Lamalle,et al.  Neural correlates of phonetic convergence and speech imitation , 2011, Front. Psychol..

[29]  Karl J. Friston,et al.  Attentional Modulation of Alpha/Beta and Gamma Oscillations Reflect Functionally Distinct Processes , 2014, The Journal of Neuroscience.

[30]  Yasuhiro Katagiri,et al.  Prosodic alignment in human–computer interaction , 2007, Connect. Sci..

[31]  Jennifer S. Pardo,et al.  On phonetic convergence during conversational interaction. , 2006, The Journal of the Acoustical Society of America.

[32]  M. Pickering,et al.  An integrated theory of language production and comprehension. , 2013, The Behavioral and brain sciences.

[33]  Michael L. Anderson Neural reuse: A fundamental organizational principle of the brain , 2010, Behavioral and Brain Sciences.

[34]  Miles A. Whittington,et al.  Top-Down Beta Rhythms Support Selective Attention via Interlaminar Interaction: A Model , 2013, PLoS Comput. Biol..

[35]  Y. Paulignan,et al.  Neural correlates of non-verbal social interactions: A dual-EEG study , 2014, Neuropsychologia.

[36]  F. L. D. Silva,et al.  Event-related EEG/MEG synchronization and desynchronization: basic principles , 1999, Clinical Neurophysiology.

[37]  Martin J. Pickering,et al.  Toward a neural basis of interactive alignment in conversation , 2012, Front. Hum. Neurosci..

[38]  Luciano Fadiga,et al.  Motor cortex compensates for lack of sensory and motor experience during auditory speech perception , 2019, Neuropsychologia.

[39]  M. Natale CONVERGENCE OF MEAN VOCAL INTENSITY IN DYADIC COMMUNICATION AS A FUNCTION OF SOCIAL DESIRABILITY , 1975 .

[40]  R. Aslin,et al.  Preference for infant-directed speech in the first month after birth. , 1990, Child development.

[41]  R. Hari Action-perception connection and the cortical mu rhythm. , 2006, Progress in brain research.

[42]  Haizhou Li,et al.  An overview of text-independent speaker recognition: From features to supervectors , 2010, Speech Commun..

[43]  Jennifer S. Pardo,et al.  Phonetic convergence across multiple measures and model talkers , 2016, Attention, Perception, & Psychophysics.

[44]  Y. Yamaguchi,et al.  Inter-brain synchronization during coordination of speech rhythm in human-to-human social interaction , 2013, Scientific Reports.

[45]  Qian Wang,et al.  Selective Attention Enhances Beta-Band Cortical Oscillation to Speech under “Cocktail-Party” Listening Conditions , 2017, Front. Hum. Neurosci..

[46]  M. Hallett,et al.  Event-related coherence and event-related desynchronization/synchronization in the 10 Hz and 20 Hz EEG during self-paced movements. , 1997, Electroencephalography and clinical neurophysiology.

[47]  Karl J. Friston,et al.  Action understanding and active inference , 2011, Biological Cybernetics.

[48]  L D Braida,et al.  Intelligibility of conversational and clear speech in noise and reverberation for listeners with normal and impaired hearing. , 1994, The Journal of the Acoustical Society of America.

[49]  S. Garrod,et al.  Brain-to-brain coupling: a mechanism for creating and sharing a social world , 2012, Trends in Cognitive Sciences.

[50]  D. Pisoni,et al.  SPEECH PERCEPTION AS A TALKER-CONTINGENT PROCESS. , 1993, Psychological science.

[51]  David Poeppel,et al.  Cortical oscillations and speech processing: emerging computational principles and operations , 2012, Nature Neuroscience.

[52]  Jennifer S. Pardo,et al.  Conversational role influences speech imitation. , 2010, Attention, perception & psychophysics.

[53]  D. Poeppel,et al.  Coupled neural systems underlie the production and comprehension of naturalistic narrative speech , 2014, Proceedings of the National Academy of Sciences.

[54]  S. Cochin,et al.  Observation and execution of movement: similarities demonstrated by quantified electroencephalography , 1999, The European journal of neuroscience.

[55]  A. Harkrider,et al.  Temporal dynamics of sensorimotor integration in speech perception and production: independent component analysis of EEG data , 2014, Front. Psychol..

[56]  Lars Kai Hansen,et al.  Frontal alpha oscillations distinguish leaders from followers: Multivariate decoding of mutually interacting brains , 2014, NeuroImage.

[57]  Luciano Fadiga,et al.  The role of the motor system in discriminating normal and degraded speech sounds , 2012, Cortex.

[58]  M. Bastiaansen,et al.  Fast oscillatory dynamics during language comprehension: Unification versus maintenance and prediction? , 2015, Brain and Language.

[59]  Chaozhe Zhu,et al.  Neural Synchronization during Face-to-Face Communication , 2012, The Journal of Neuroscience.

[60]  Nick Campbell,et al.  Investigating automatic measurements of prosodic accommodation and its dynamics in social interaction , 2014, Speech Commun..

[61]  Jennifer S. Pardo,et al.  Measuring phonetic convergence in speech production , 2013, Front. Psychol..

[62]  D. Poeppel,et al.  The cortical organization of speech processing , 2007, Nature Reviews Neuroscience.

[63]  Andrea Gaggioli,et al.  Automatic imitation of the arm kinematic profile in interacting partners , 2015, Cognitive Processing.

[64]  J. Kelso,et al.  The coordination dynamics of social neuromarkers , 2013, Front. Hum. Neurosci..

[65]  Ivan Toni,et al.  Conceptual Alignment: How Brains Achieve Mutual Understanding , 2016, Trends in Cognitive Sciences.

[66]  John-Dylan Haynes,et al.  Content-specific coordination of listeners' to speakers' EEG during communication , 2012, Front. Hum. Neurosci..

[67]  Margarita Kaushanskaya,et al.  The Language Experience and Proficiency Questionnaire (LEAP-Q): assessing language profiles in bilinguals and multilinguals. , 2007, Journal of speech, language, and hearing research : JSLHR.

[68]  Matthew H. Davis,et al.  Evidence for causal top-down frontal contributions to predictive processes in speech perception , 2017, Nature Communications.

[69]  L. Fadiga,et al.  The Motor Somatotopy of Speech Perception , 2009, Current Biology.

[70]  M. Pickering,et al.  Prediction and imitation in speech , 2013, Front. Psychol..

[71]  Larry P. Heck,et al.  MSR Identity Toolbox v1.0: A MATLAB Toolbox for Speaker Recognition Research , 2013 .

[72]  Gregor Thut,et al.  Lip movements entrain the observers’ low-frequency brain oscillations to facilitate speech intelligibility , 2016, eLife.

[73]  Diane J. Litman,et al.  Automatically measuring lexical and acoustic/prosodic convergence in tutorial dialog corpora , 2007, SLaTE.

[74]  A. Samuel,et al.  Perceptual learning for speech , 2009, Attention, perception & psychophysics.

[75]  Giovanni Pezzulo,et al.  You cannot speak and listen at the same time: a probabilistic model of turn-taking , 2017, Biological Cybernetics.

[76]  T. Paus,et al.  Seeing and hearing speech excites the motor system involved in speech production , 2003, Neuropsychologia.

[77]  S. Jun,et al.  Interbrain phase synchronization during turn‐taking verbal interaction—a hyperscanning study using simultaneous EEG/MEG , 2018, Human brain mapping.

[78]  K. Watkins,et al.  Motor Representations of Articulators Contribute to Categorical Perception of Speech Sounds , 2009, The Journal of Neuroscience.

[79]  H. Kennedy,et al.  Visual Areas Exert Feedforward and Feedback Influences through Distinct Frequency Channels , 2014, Neuron.

[80]  Elise A. Piazza,et al.  Measuring speaker–listener neural coupling with functional near infrared spectroscopy , 2016, Scientific Reports.

[81]  Julie M. Hupp,et al.  Speech priming: Evidence for rate persistence in unscripted speech , 2009 .

[82]  Line Garnero,et al.  Inter-Brain Synchronization during Social Interaction , 2010, PloS one.

[83]  Francisco Torreira,et al.  Timing in turn-taking and its implications for processing models of language , 2015, Front. Psychol..

[84]  G. Pfurtscheller,et al.  Evaluation of event-related desynchronization (ERD) preceding and following voluntary self-paced movement. , 1979, Electroencephalography and clinical neurophysiology.

[85]  L. Turner,et al.  Understanding Interpersonal Communication: Making Choices in Changing Times , 2010 .

[86]  P. Trofimovich,et al.  Interactive alignment between bilingual interlocutors: Evidence from two information-exchange tasks* , 2014, Bilingualism: Language and Cognition.

[87]  Riitta Hari,et al.  Actor's and observer's primary motor cortices stabilize similarly after seen or heard motor actions , 2007, Proceedings of the National Academy of Sciences.

[88]  Robert Oostenveld,et al.  FieldTrip: Open Source Software for Advanced Analysis of MEG, EEG, and Invasive Electrophysiological Data , 2010, Comput. Intell. Neurosci..

[89]  David G. Norris,et al.  Combining EEG and fMRI to investigate the post-movement beta rebound , 2006, NeuroImage.

[90]  Mattias Heldner,et al.  Pause and gap length in face-to-face interaction , 2009, INTERSPEECH.

[91]  Kai Vogeley,et al.  A second-person neuroscience in interaction 1 , 2013, Behavioral and Brain Sciences.

[92]  Luciano Fadiga,et al.  Analyzing Vocal Tract Movements During Speech Accommodation , 2018, INTERSPEECH.

[93]  F. Pulvermüller Neural reuse of action perception circuits for language, concepts and communication , 2018, Progress in Neurobiology.

[94]  G. Sandini,et al.  Motor Contagion during Human-Human and Human-Robot Interaction , 2014, PloS one.

[95]  Gérard Bailly,et al.  Speech dominoes and phonetic convergence , 2010, INTERSPEECH.

[96]  Jon Andoni Duñabeitia,et al.  Brain-to-brain entrainment: EEG interbrain synchronization while speaking and listening , 2017, Scientific Reports.

[97]  B. Ross,et al.  Internalized Timing of Isochronous Sounds Is Represented in Neuromagnetic Beta Oscillations , 2012, The Journal of Neuroscience.

[98]  D. Poeppel,et al.  Phase Patterns of Neuronal Responses Reliably Discriminate Speech in Human Auditory Cortex , 2007, Neuron.

[99]  Peter Hagoort,et al.  What can we learn from a two-brain approach to verbal interaction? , 2016, Neuroscience & Biobehavioral Reviews.

[100]  Fabian Ramseyer,et al.  Nonverbal Synchrony or Random Coincidence? How to Tell the Difference , 2009, COST 2102 Training School.

[101]  Samuel Weller Singer,et al.  The Same , 1880, The Indian medical gazette.

[102]  Douglas A. Reynolds,et al.  Speaker Verification Using Adapted Gaussian Mixture Models , 2000, Digit. Signal Process..

[103]  J. Simon,et al.  Cortical entrainment to continuous speech: functional roles and interpretations , 2014, Front. Hum. Neurosci..

[104]  L. Fadiga,et al.  Listener-speaker perceived distance predicts the degree of motor contribution to speech perception. , 2015, Cerebral cortex.

[105]  R. Hari,et al.  Brain basis of human social interaction: from concepts to brain imaging. , 2009, Physiological reviews.

[106]  N. George,et al.  Oscillatory Brain Correlates of Live Joint Attention: A Dual-EEG Study , 2012, Front. Hum. Neurosci..

[107]  A. Engel,et al.  Beta-band oscillations—signalling the status quo? , 2010, Current Opinion in Neurobiology.

[108]  E. Ferrari,et al.  Listening to speech recruits specific tongue motor synergies as revealed by transcranial magnetic stimulation and tissue-Doppler ultrasound imaging , 2014, Philosophical Transactions of the Royal Society B: Biological Sciences.

[109]  Björn Lindblom,et al.  Explaining Phonetic Variation: A Sketch of the H&H Theory , 1990 .

[110]  Gérard Bailly,et al.  Assessing objective characterizations of phonetic convergence , 2014, INTERSPEECH.

[111]  Tessa Bent,et al.  Perceptual adaptation to non-native speech , 2008, Cognition.

[112]  T. Gollan,et al.  Self-ratings of spoken language dominance: A Multilingual Naming Test (MINT) and preliminary norms for young and aging Spanish–English bilinguals* , 2011, Bilingualism: Language and Cognition.

[113]  U. Hasson,et al.  Speaker–listener neural coupling underlies successful communication , 2010, Proceedings of the National Academy of Sciences.

[114]  G. Rizzolatti,et al.  Speech listening specifically modulates the excitability of tongue muscles: a TMS study , 2002, The European journal of neuroscience.

[115]  C. Schroeder,et al.  Low-frequency neuronal oscillations as instruments of sensory selection , 2009, Trends in Neurosciences.