Effects of separating auditory and visual sources on audiovisual integration of speech

When the image of a speaker saying the bisyllable /aga/ is presented in synchrony with the sound of a speaker saying /aba/, subjects tend to report hearing the sound /ada/. The present experiment explores the effects of spatial separation on this class of perceptual illusion known as the McGurk effect. Synchronous auditory and visual speech signals were presented from different locations. The auditory signal was presented from positions 0°, 30°, 60° and 90° in azimuth away from the visual signal source. The results show that spatial incongruencies do not substantially influence the multimodal integration of speech signals.

[1]  A G Samuel,et al.  Phonetic prototypes , 1982, Perception & psychophysics.

[2]  P. Kuhl.  Human adults and human infants show a “perceptual magnet effect” for the prototypes of speech categories, monkeys do not , 1991, Perception & psychophysics.

[3]  H. McGurk,et al.  Visual influences on speech perception processes , 1978, Perception & psychophysics.

[4]  L. Rosenblum,et al.  An audiovisual test of kinematic primitives for visual speech perception. , 1996, Journal of experimental psychology. Human perception and performance.

[5]  P. Gribble,et al.  Temporal constraints on the McGurk effect , 1996, Perception & psychophysics.

[6]  The cognitive architecture of bimodal event perception: A commentary and addendum to Radeau (1994) , 1994 .

[7]  W R Thurlow,et al.  Effects of degree of visual association and angle of displacement on the "ventriloquism" effect. , 1973, Perceptual and motor skills.

[8]  Béatrice de Gelder,et al.  Exploring the relation between mcgurk interference and ventriloquism , 1994, ICSLP.

[9]  Bruno H. Repp,et al.  Exploring the “McGurk effect” , 1983 .

[10]  P. Bertelson,et al.  Adaptation to auditory-visual discordance and ventriloquism in semirealistic situations , 1977 .

[11]  P K Kuhl,et al.  The role of visual information in the processing of , 1989, Perception & psychophysics.

[12]  Q Summerfield,et al.  Use of Visual Information for Phonetic Perception , 1979, Phonetica.

[13]  P. Bertelson,et al.  Cross-modal bias and perceptual fusion with auditory-visual spatial discordance , 1981, Perception & psychophysics.

[14]  Q Summerfield,et al.  Detection and Resolution of Audio-Visual Incompatibility in the Perception of Vowels , 1984, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[15]  P Bertelson,et al.  Cognitive factors and adaptation to auditory-visual discordance , 1978, Perception & psychophysics.

[16]  J L Miller,et al.  A possible auditory basis for internal structure of phonetic categories. , 1983, The Journal of the Acoustical Society of America.

[17]  Q. Summerfield.  Some preliminaries to a comprehensive account of audio-visual speech perception. , 1987 .

[18]  D W Massaro,et al.  American Psychological Association, Inc. Evaluation and Integration of Visual and Auditory Information in Speech Perception , 2022 .

[19]  D. H. Warren,et al.  Immediate perceptual response to intersensory discrepancy. , 1980, Psychological bulletin.

[20]  J. Gibson.  The Senses Considered As Perceptual Systems , 1967 .

[21]  B. Walden,et al.  Effects of training on the visual recognition of consonants. , 1977, Journal of speech and hearing research.

[22]  M. Radeau.  Auditory-visual spatial interaction and modularity , 1994, Current psychology of cognition = Cahiers de psychologie cognitive : CPC.

[23]  D. Massaro.  Speech Perception By Ear and Eye: A Paradigm for Psychological Inquiry , 1989 .

[24]  D W Massaro,et al.  Perception of asynchronous and conflicting visual and auditory speech. , 1996, The Journal of the Acoustical Society of America.

[25]  Y. Tohkura,et al.  McGurk effect in non-English listeners: few visual effects for Japanese subjects hearing Japanese syllables of high auditory intelligibility. , 1991, The Journal of the Acoustical Society of America.

[26]  Lawrence D. Rosenblum,et al.  An audiovisual test of kinematic primitives for visual speech perception. , 1996 .

[27]  G. A. Miller,et al.  The intelligibility of speech as a function of the context of the test materials. , 1951, Journal of experimental psychology.

[28]  R. Campbell,et al.  Hearing by eye : the psychology of lip-reading , 1988 .

[29]  P. Kuhl,et al.  Integral processing of visual place and auditory voicing information during phonetic perception. , 1991, Journal of experimental psychology. Human perception and performance.

[30]  Andrew N. Meltzoff,et al.  Factors affecting the integration of auditory and visual information in speech: The level effect , 1988 .

[31]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[32]  M. Mendelson,et al.  Acoustic-optical correspondences and auditory-visual coordination in infancy. , 1979, Canadian journal of psychology.

[33]  Y. Tohkura,et al.  Inter-language differences in the influence of visual cues in speech perception. , 1993 .

[34]  R. Lynn,et al.  A secular decline in the strength of Spearman’s g in Japan , 1994 .

[35]  Antoinette T. Gesi,et al.  Bimodal speech perception: an examination across languages , 1993 .

[36]  Vicki Bruce,et al.  Facial identity and facial speech processing: Familiar faces and voices in the McGurk effect , 1995, Perception & psychophysics.

[37]  W. H. Sumby,et al.  Visual contribution to speech intelligibility in noise , 1954 .

[38]  R B Welch,et al.  Effect of Degree of Separation of Visual-Auditory Stimulus and Eye Position upon Spatial Interaction of Vision and Audition , 1976, Perceptual and motor skills.