Audiovisual gating and the time course of speech perception.

The time course of audiovisual information in speech perception was examined using a gating paradigm. VCVs that evoked the McGurk effect were gated visually and auditorily. The visual gating yielded a McGurk effect that increased in strength as a linear function of amount of visual stimulus presented. The acoustic gating revealed a more nonlinear function in which the VC information was considerably weaker than the CV portion of the VCV. The results suggest that the flow of cross-modal information in quite complex during audiovisual speech perception.

[1]  Dominic W. Massaro,et al.  Perceiving asynchronous bimodal speech in consonant-vowel and vowel syllables , 1993, Speech Commun..

[2]  Andrew N. Meltzoff,et al.  Factors affecting the integration of auditory and visual information in speech: The effect of vowel environment , 1988 .

[3]  D. Shankweiler,et al.  What information enables a listener to map a talker's vowel space? , 1976, The Journal of the Acoustical Society of America.

[4]  F. Grosjean,et al.  The gating paradigm: A comparison of successive and individual presentation formats , 1984 .

[5]  Mingzhou Ding,et al.  Evaluation of a dynamical model of speech perception , 1995 .

[6]  S. Blumstein,et al.  Perceptual invariance and onset spectra for stop consonants in different vowel environments. , 1980, The Journal of the Acoustical Society of America.

[7]  D. H. Warren,et al.  Immediate perceptual response to intersensory discrepancy. , 1980, Psychological bulletin.

[8]  H. McGurk,et al.  Visual influences on speech perception processes , 1978, Perception & psychophysics.

[9]  P K Kuhl,et al.  The role of visual information in the processing of , 1989, Perception & psychophysics.

[10]  A. Meltzoff,et al.  Integrating speech information across talkers, gender, and sensory modality: Female faces and male voices in the McGurk effect , 1991, Perception & psychophysics.

[11]  V. Mann,et al.  Influence of vocalic context on perception of the [∫]-[s] distinction , 1978 .

[12]  R. Hari,et al.  Seeing speech: visual information from lip movements modifies activity in the human auditory cortex , 1991, Neuroscience Letters.

[13]  K. Green.  The perception of speaking rate using visual information from a talker’s face , 1987, Perception & psychophysics.

[14]  D. Pisoni,et al.  Speech perception without traditional speech cues. , 1981, Science.

[15]  A M Liberman,et al.  Perception of the speech code. , 1967, Psychological review.

[16]  A. Liberman,et al.  Some experiments on the sound of silence in phonetic perception. , 1979, The Journal of the Acoustical Society of America.

[17]  Q Summerfield,et al.  Detection and Resolution of Audio-Visual Incompatibility in the Perception of Vowels , 1984, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[18]  F Grosjean,et al.  Spoken word recognition processes and the gating paradigm , 1980, Perception & psychophysics.

[19]  Dani Byrd.  Perception of Assimilation in Consonants Clusters:A Gestural Model , 1992 .

[20]  Q. Summerfield.  Some preliminaries to a comprehensive account of audio-visual speech perception. , 1987 .

[21]  J. Kelso,et al.  Evaluation of a dynamical model of speech perception. , 1994, Perception & psychophysics.

[22]  L K Tyler,et al.  Is gating an on-line task? Evidence from naming latency data , 1985, Perception & psychophysics.

[23]  S. Öhman.  Numerical Model of Coarticulation , 1967 .

[24]  N. F. Dixon,et al.  The Detection of Auditory Visual Desynchrony , 1980, Perception.

[25]  A. Meltzoff,et al.  Investigating the role of specific facial information in audio‐visual speech perception , 1995 .

[26]  K. Green,et al.  Cross-modal discrepancies in coarticulation and the integration of speech information: the McGurk effect with mismatched vowels. , 1995, Journal of experimental psychology. Human perception and performance.

[27]  Vincent J. van Heuven,et al.  Intelligibility of audio-visually desynchronised speech: asymmetrical effect of phoneme position , 1992, ICSLP.

[28]  Q. Summerfield,et al.  Lipreading and audio-visual speech perception. , 1992, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[29]  P. Smeele,et al.  Perceiving speech: Integrating auditory and visual speech , 1994 .

[30]  Y. Tohkura,et al.  McGurk effect in non-English listeners: few visual effects for Japanese subjects hearing Japanese syllables of high auditory intelligibility. , 1991, The Journal of the Acoustical Society of America.

[31]  S. Ohman.  Numerical model of coarticulation. , 1967, The Journal of the Acoustical Society of America.

[32]  V L Gracco,et al.  Timing factors in the coordination of speech movements , 1988, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[33]  P. Gribble,et al.  Temporal constraints on the McGurk effect , 1996, Perception & psychophysics.

[34]  Eric Vatikiotis-Bateson,et al.  The moving face during speech communication , 1998 .

[35]  J. L. Miller,et al.  On the role of visual rate information in phonetic perception , 1985, Perception & psychophysics.

[36]  L A Streeter,et al.  The role of medial consonant transitions in word perception. , 1979, The Journal of the Acoustical Society of America.

[37]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[38]  Anne Cutler,et al.  Spoken word recognition and production , 1995 .

[39]  D. Shankweiler,et al.  What information enables a listener to map a talker's vowel space? , 1974, The Journal of the Acoustical Society of America.

[40]  Bruno H. Repp,et al.  Exploring the “McGurk effect” , 1983 .

[41]  W. H. Sumby,et al.  Visual contribution to speech intelligibility in noise , 1954 .

[42]  W. Strange.  Evolving theories of vowel perception. , 1987, The Journal of the Acoustical Society of America.

[43]  Christian Abry,et al.  Does movement on the lips mean movement in the mind , 1996 .

[44]  Fredericka Bell‐Berti,et al.  A Temporal Model of Speech Production , 1980, Phonetica.

[45]  Hani Yehia,et al.  Characterizing audiovisual information during speech , 1996, Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96.