Audio-visual Speech Processing

Speech has visual as well as auditory correlates. What does this imply for the perception of speech and its neural bases? The perception of speech by ear and by eye makes use of mechanisms that are inherently multimodal, concerned with speech as an activity that has, at the same time, motor, acoustic, visual, and somaesthetic properties that enter into its representation and processing in a coherent and integrated way. The challenge for speech perception theories is to encompass these properties appropriately at neurological, computational, and psychological levels.

[1]  Dominic W. Massaro,et al.  Animated speech: research progress and applications , 2001, AVSP.

[2]  M A Goodale,et al.  Dynamic visual speech perception in a patient with visual form agnosia , 2002, Neuroreport.

[3]  L D Rosenblum,et al.  Selective adaptation in speech perception using a compelling audiovisual adaptor. , 1994, The Journal of the Acoustical Society of America.

[4]  J. Driver.  Enhancement of selective listening by illusory mislocation of speech sounds due to lip-reading , 1996, Nature.

[5]  M M Cohen,et al.  Speechreading in the akinetopsic patient, L.M. , 1997, Brain : a journal of neurology.

[6]  P. Deltenre,et al.  Mismatch negativity evoked by the McGurk–MacDonald effect: a phonetic representation within short-term memory , 2002, Clinical Neurophysiology.

[7]  R. Campbell,et al.  Hearing by Eye , 1980, The Quarterly journal of experimental psychology.

[8]  Jeffery A. Jones,et al.  Multisensory Integration Sites Identified by Perception of Spatial Wavelet Filtered Visual Speech Gesture Information , 2004, Journal of Cognitive Neuroscience.

[9]  Q. Summerfield,et al.  Intermodal timing relations and audio-visual speech recognition by normal-hearing adults. , 1985, The Journal of the Acoustical Society of America.

[10]  P Bertelson,et al.  Cognitive factors and adaptation to auditory-visual discordance , 1978, Perception & psychophysics.

[11]  Jeffery A. Jones,et al.  Visual Prosody and Speech Intelligibility , 2004, Psychological science.

[12]  Veikko Jousmäki,et al.  MEG studies of gross-modal integration and plasticity , 2004 .

[13]  A. Liberman,et al.  The motor theory of speech perception revised , 1985, Cognition.

[14]  Q Summerfield,et al.  Audiovisual presentation demonstrates that selective adaptation in speech perception is purely auditory , 1981, Perception & psychophysics.

[15]  D. Pisoni,et al.  Cross-modal source information and spoken word recognition. , 2004, Journal of experimental psychology. Human perception and performance.

[16]  Vicki Bruce,et al.  Facial identity and facial speech processing: Familiar faces and voices in the McGurk effect , 1995, Perception & psychophysics.

[17]  Jeffery A. Jones,et al.  Neural processes underlying perceptual enhancement by visual speech gestures , 2003, Neuroreport.

[18]  L. Rosenblum,et al.  The McGurk effect in infants , 1997 .

[19]  E. Vatikiotis-Bateson,et al.  `Putting the Face to the Voice' Matching Identity across Modality , 2003, Current Biology.

[20]  P F Seitz,et al.  The use of visible speech cues for improving auditory detection of spoken sentences. , 2000, The Journal of the Acoustical Society of America.

[21]  E. Bullmore,et al.  Activation of auditory cortex during silent lipreading. , 1997, Science.

[22]  P. Gribble,et al.  Temporal constraints on the McGurk effect , 1996, Perception & psychophysics.

[23]  J. Schwartz,et al.  Seeing to hear better: evidence for early audio-visual interactions in speech identification , 2004, Cognition.

[24]  L. Bernstein,et al.  Audiovisual Speech Binding: Convergence or Association? , 2004 .

[25]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[26]  T. Paus,et al.  Seeing and hearing speech excites the motor system involved in speech production , 2003, Neuropsychologia.

[27]  M. Sams,et al.  Time course of multisensory interactions during audiovisual speech perception in humans: a magnetoencephalographic study , 2004, Neuroscience Letters.

[28]  Jeesun Kim,et al.  Hearing Foreign Voices: Does Knowing What is Said Affect Visual-Masked-Speech Detection? , 2003, Perception.