Seeing and hearing others and oneself talk.

We studied the modification of auditory perception in three different conditions in twenty subjects. Observing other person's discordant articulatory gestures deteriorated identification of acoustic speech stimuli and modified the auditory percept, causing a strong McGurk effect. A similar effect was found when the subjects watched their own silent articulation in a mirror and acoustic stimuli were simultaneously presented to their ears. Interestingly, a smaller but significant effect was even obtained when the subjects just silently articulated the syllables without visual feedback. On the other hand, observing other person's or one's own concordant articulation and silently articulating a concordant syllable improved identification of the acoustic stimuli. The modification of auditory percepts caused by visual observation of speech and silently articulating it are both suggested to be due to the alteration of activity in the auditory cortex. Our findings support the idea of a close relationship between speech perception and production.

[1]  M. Jeannerod.  TO ACT OR NOT TO ACT : PERSPECTIVES ON THE REPRESENTATION OF ACTIONS , 1999 .

[2]  H Bekkering,et al.  Motor activation from visible speech: evidence from stimulus response compatibility. , 2000, Journal of experimental psychology. Human perception and performance.

[3]  G. Rizzolatti,et al.  Mirror neurons responding to the observation of ingestive and communicative mouth actions in the monkey ventral premotor cortex , 2003, The European journal of neuroscience.

[4]  Robert Chen,et al.  Observation–execution matching system for speech: a magnetic stimulation study , 2001, Neuroreport.

[5]  M. Merzenich,et al.  Modulation of the Auditory Cortex during Speech: An MEG Study , 2002, Journal of Cognitive Neuroscience.

[6]  M. Arbib,et al.  Language within our grasp , 1998, Trends in Neurosciences.

[7]  N. A. Borghese,et al.  A functional-anatomical model for lipreading. , 2003, Journal of neurophysiology.

[8]  M. Jeannerod.  The 25th Bartlett Lecture , 1999 .

[9]  Mikko Sams,et al.  McGurk effect in Finnish syllables, isolated words, and words in sentences: Effects of word meaning and sentence context , 1998, Speech Commun..

[10]  J. Mazziotta,et al.  Cortical mechanisms of human imitation. , 1999, Science.

[11]  Z Al-Kawi,et al.  Book review: Brain topography today. , 1998, Annals of Saudi medicine.

[12]  Kevin G. Munhall,et al.  Effects of separating auditory and visual sources on audiovisual integration of speech , 1997 .

[13]  G. Curio,et al.  Speaking modifies voice‐evoked activity in the human auditory cortex , 2000, Human brain mapping.

[14]  R. Hari,et al.  Seeing speech: visual information from lip movements modifies activity in the human auditory cortex , 1991, Neuroscience Letters.

[15]  H. Fukuyama,et al.  Cortical processing mechanism for vocalization with auditory verbal feedback , 1997, Neuroreport.

[16]  W. Gantt.  Biology and Neurophysiology of the Conditioned Reflex and Its Role in Adaptive Behavior , 1976 .

[17]  W. H. Sumby,et al.  Visual contribution to speech intelligibility in noise , 1954 .

[18]  T. Paus,et al.  Seeing and hearing speech excites the motor system involved in speech production , 2003, Neuropsychologia.

[19]  C. Fowler,et al.  Listening with eye and hand: Cross-modal contributions to speech perception. , 1991 .

[20]  C. Price.  The functional anatomy of word comprehension and production , 1998, Trends in Cognitive Sciences.

[21]  E. Holst,et al.  Das Reafferenzprinzip , 2004, Naturwissenschaften.

[22]  E. Bullmore,et al.  Activation of auditory cortex during silent lipreading. , 1997, Science.

[23]  Mikko Sams,et al.  A neuromagnetic study of the integration of audiovisual speech in the brain , 1997 .

[24]  J. O'neill.  Contributions of the visual components of oral symbols to speech comprehension. , 1954, The Journal of speech and hearing disorders.

[25]  Riitta Salmelin,et al.  Subject's own speech reduces reactivity of the human auditory cortex , 1999, Neuroscience Letters.

[26]  R. Hari,et al.  Viewing Lip Forms Cortical Dynamics , 2002, Neuron.

[27]  Gabriel Curio,et al.  Differential effects of overt, covert and replayed speech on vowel-evoked responses of the human auditory cortex , 1999, Neuroscience Letters.

[28]  Mikko Sams,et al.  Processing of changes in visual speech in the human auditory cortex. , 2002, Brain research. Cognitive brain research.

[29]  R. Hari,et al.  Temporal dynamics of cortical representation for action. , 2000, Proceedings of the National Academy of Sciences of the United States of America.

[30]  A. Mechelli,et al.  Neuroimaging Studies of Word and Pseudoword Reading: Consistencies, Inconsistencies, and Limitations , 2003, Journal of Cognitive Neuroscience.

[31]  G. Rizzolatti,et al.  Speech listening specifically modulates the excitability of tongue muscles: a TMS study , 2002, The European journal of neuroscience.

[32]  P. Arnold,et al.  Bisensory augmentation: a speechreading advantage when speech is clearly audible and intact. , 2001, British journal of psychology.

[33]  Alan C. Evans,et al.  Modulation of cerebral blood-flow in the human auditory cortex during speech: role of motor-to-sensory discharges , 1996, NeuroImage.

[34]  Sean Marrett,et al.  Imaging Motor-to-Sensory Discharges in the Human Brain: An Experimental Tool for the Assessment of Functional Connectivity , 1996, NeuroImage.

[35]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[36]  P. McGuire,et al.  Cortical substrates for the perception of face actions: an fMRI study of the specificity of activation for seen speech and for meaningless lower-face acts (gurning). , 2001, Brain research. Cognitive brain research.

[37]  E. Bullmore,et al.  Response amplification in sensory-specific cortices during crossmodal binding. , 1999, Neuroreport.

[38]  L. Bernstein,et al.  Visual speech perception without primary auditory cortex activation , 2002, Neuroreport.

[39]  Colin M. Brown,et al.  The Neural Circuitry Involved in the Reading of German Words and Pseudowords: A PET Study , 1999, Journal of Cognitive Neuroscience.

[40]  A. Liberman,et al.  The motor theory of speech perception revised , 1985, Cognition.

[41]  Mikko Sams,et al.  Audio-visual speech synthesis for finnish , 1999, AVSP.