Temporal constraints on the McGurk effect

Three experiments are reported on the influence of different timing relations on the McGurk effect. In the first experiment, it is shown that strict temporal synchrony between auditory and visual speech stimuli is not required for the McGurk effect. Subjects were strongly influenced by the visual stimuli when the auditory stimuli lagged the visual stimuli by as much as 180 msec. In addition, a stronger McGurk effect was found when the visual and auditory vowels matched. In the second experiment, we paired auditory and visual speech stimuli produced under different speaking conditions (fast, normal, clear). The results showed that the manipulations in both the visual and auditory speaking conditions independently influenced perception. In addition, there was a small but reliable tendency for the better matched stimuli to elicit more McGurk responses than unmatched conditions. In the third experiment, we combined auditory and visual stimuli produced under different speaking conditions (fast, clear) and delayed the acoustics with respect to the visual stimuli. The subjects showed the same pattern of results as in the second experiment. Finally, the delay did not cause different patterns of results for the different audiovisual speaking style combinations. The results suggest that perceivers may be sensitive to the concordance of the time-varying aspects of speech but they do not require temporal coincidence of that information.

[1]  W. H. Sumby,et al.  Visual contribution to speech intelligibility in noise , 1954 .

[2]  C. Dunnett.  A Multiple Comparison Procedure for Comparing Several Treatments with a Control , 1955 .

[3]  I. Hirsh,et al.  Perceived order in different sense modalities. , 1961, Journal of experimental psychology.

[4]  S. Öhman.  Numerical Model of Coarticulation , 1967 .

[5]  S. Ohman.  Numerical model of coarticulation. , 1967, The Journal of the Acoustical Society of America.

[6]  A M Liberman,et al.  Perception of the speech code. , 1967, Psychological review.

[7]  D. H. Warren,et al.  Sensory conflict in judgments of spatial direction , 1969 .

[8]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[9]  H. McGurk,et al.  Visual influences on speech perception processes , 1978, Perception & psychophysics.

[10]  N. F. Dixon,et al.  The Detection of Auditory Visual Desynchrony , 1980, Perception.

[11]  D. H. Warren,et al.  Immediate perceptual response to intersensory discrepancy. , 1980, Psychological bulletin.

[12]  T. Gay.  Mechanisms in the Control of Speech Rate , 1981, Phonetica.

[13]  Bruno H. Repp,et al.  Exploring the “McGurk effect” , 1983 .

[14]  J L Miller,et al.  Some effects of speaking rate on the production of /b/ and /w/. , 1983, The Journal of the Acoustical Society of America.

[15]  Q Summerfield,et al.  Detection and Resolution of Audio-Visual Incompatibility in the Perception of Vowels , 1984, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[16]  B. Green.  Heat pain thresholds in the oral-facial region , 1985, Perception & psychophysics.

[17]  N I Durlach,et al.  Speaking clearly for the hard of hearing I: Intelligibility differences between clear and conversational speech. , 1985, Journal of speech and hearing research.

[18]  Q. Summerfield,et al.  Intermodal timing relations and audio-visual speech recognition by normal-hearing adults. , 1985, The Journal of the Acoustical Society of America.

[19]  A. Ellis.  Progress in the psychology of language , 1985 .

[20]  J. L. Miller,et al.  On the role of visual rate information in phonetic perception , 1985, Perception & psychophysics.

[21]  H Kunov,et al.  Disruptive effects of auditory signal delay on speech perception with lipreading. , 1986, The Journal of auditory research.

[22]  Robert B. Welch,et al.  Contributions of audition and vision to temporal rate perception , 1986, Perception & psychophysics.

[23]  Christian Abry,et al.  "Laws" for lips , 1986, Speech Commun..

[24]  Q. Summerfield.  Some preliminaries to a comprehensive account of audio-visual speech perception. , 1987 .

[25]  K. Green.  The perception of speaking rate using visual information from a talker’s face , 1987, Perception & psychophysics.

[26]  Andrew N. Meltzoff,et al.  Factors affecting the integration of auditory and visual information in speech: The effect of vowel environment , 1988 .

[27]  R. Campbell,et al.  Hearing by eye : the psychology of lip-reading , 1988 .

[28]  Jean Vroomen,et al.  Speech-perception by ear and eye: a paradigm for psychological inquiry. Commentary/Massaro , 1989 .

[29]  P K Kuhl,et al.  The role of visual information in the processing of , 1989, Perception & psychophysics.

[30]  Björn Lindblom,et al.  Explaining Phonetic Variation: A Sketch of the H&H Theory , 1990 .

[31]  Phillip J. Cheetham,et al.  Motion measurement using the Peak Performance Technologies system , 1990, Other Conferences.

[32]  L. Braida.  Crossmodal Integration in the Identification of Consonant Segments , 1991, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[33]  A. Meltzoff,et al.  Integrating speech information across talkers, gender, and sensory modality: Female faces and male voices in the McGurk effect , 1991, Perception & psychophysics.

[34]  C. Fowler,et al.  Listening with eye and hand: cross-modal contributions to speech perception. , 1991, Journal of experimental psychology. Human perception and performance.

[35]  Y. Tohkura,et al.  McGurk effect in non-English listeners: few visual effects for Japanese subjects hearing Japanese syllables of high auditory intelligibility. , 1991, The Journal of the Acoustical Society of America.

[36]  P. Kuhl,et al.  Integral processing of visual place and auditory voicing information during phonetic perception. , 1991, Journal of experimental psychology. Human perception and performance.

[37]  Vincent J. van Heuven,et al.  Intelligibility of audio-visually desynchronised speech: asymmetrical effect of phoneme position , 1992, ICSLP.

[38]  Q. Summerfield,et al.  Lipreading and audio-visual speech perception. , 1992, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[39]  Dominic W. Massaro,et al.  Perceiving asynchronous bimodal speech in consonant-vowel and vowel syllables , 1993, Speech Commun..

[40]  Makio Kashino,et al.  The influence of knowledge and experience during the processing of spoken words: non-native listeners , 1994, ICSLP.

[41]  P. Barber,et al.  Effect of video frame rate on subjects' ability to shadow one of two competing verbal passages. , 1994, Journal of speech and hearing research.

[42]  Vincent J. van Heuven,et al.  Temporal organization of bimodal speech information , 1994, ICSLP.

[43]  P K Kuhl,et al.  Talker continuity and the use of rate information during phonetic perception , 1994, Perception & psychophysics.

[44]  Eric Vatikiotis-Bateson,et al.  Listener eye movement behavior during audiovisual speech perception , 1994, ICSLP.

[45]  K P Green,et al.  Cross-modal discrepancies in coarticulation and the integration of speech information: the McGurk effect with mismatched vowels. , 1995, Journal of experimental psychology. Human perception and performance.

[46]  D W Massaro,et al.  Perception of asynchronous and conflicting visual and auditory speech. , 1996, The Journal of the Acoustical Society of America.