McGurk effect in Finnish syllables, isolated words, and words in sentences: Effects of word meaning and sentence context

Abstract The “McGurk effect” is a robust illusion in which subject's perception of an acoustical syllable is modified by the view of the talker's articulation. This effect is not perceived universally but, however, is experienced by the majority of the subjects. For example, if the acoustical syllable /ba/ is presented in synchrony with a face articulating /ga/, English-speaking subjects typically perceive /da/ and less frequently /ga/. We studied the McGurk effect in Finnish syllables, isolated words, and words presented in sentence context in 65 subjects. Audiovisual combinations expected to be perceived either as meaningful words or nonwords were used. Words were also presented in various positions of three-word sentences in which the expected word could match or mismatch with the sentence context. A strong McGurk effect was obtained with each stimulus type. In addition, the strength of the McGurk effect did not appear to be influenced by word meaning or sentence context. These findings support the idea that audiovisual speech integration occurs at phonetic perceptual level before the word meaning is extracted.

[1]  A. Samuel.  The role of bottom-up confirmation in the phonemic restoration illusion. , 1981, Journal of experimental psychology. Human perception and performance.

[2]  P. L. Jackson.  The Theoretical Minimal Unit for Visual Speech Perception: Visemes and Coarticulation. , 1988 .

[3]  W. H. Sumby,et al.  Visual contribution to speech intelligibility in noise , 1954 .

[4]  D. Massaro.  Speech Perception By Ear and Eye: A Paradigm for Psychological Inquiry , 1989 .

[5]  R. Ilmoniemi,et al.  Magnetoencephalography-theory, instrumentation, and applications to noninvasive studies of the working human brain , 1993 .

[6]  D. Reisberg,et al.  Easy to hear but hard to understand: A lip-reading advantage with intact auditory stimuli. , 1987 .

[7]  Mikko Sams,et al.  A neuromagnetic study of the integration of audiovisual speech in the brain , 1997 .

[8]  E. Bullmore,et al.  Activation of auditory cortex during silent lipreading. , 1997, Science.

[9]  A. Meltzoff,et al.  Integrating speech information across talkers, gender, and sensory modality: Female faces and male voices in the McGurk effect , 1991, Perception & psychophysics.

[10]  Kevin G. Munhall,et al.  Effects of separating auditory and visual sources on audiovisual integration of speech , 1997 .

[11]  G. A. Miller,et al.  An Analysis of Perceptual Confusions Among Some English Consonants , 1955 .

[12]  J. L. Miller,et al.  On the role of visual rate information in phonetic perception , 1985, Perception & psychophysics.

[13]  M. Funnell,et al.  Audiovisual integration in perception of real words , 1992, Perception & psychophysics.

[14]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[15]  W. Ganong.  Phonetic categorization in auditory word perception. , 1980, Journal of experimental psychology. Human perception and performance.

[16]  P. Kricos.  Differences in Visual Intelligibility Across Talkers , 1996 .

[17]  A A Montgomery,et al.  Auditory and visual contributions to the perception of consonants. , 1974, Journal of speech and hearing research.

[18]  Y. Tohkura,et al.  Inter-language differences in the influence of visual cues in speech perception. , 1993 .

[19]  L D Rosenblum,et al.  Selective adaptation in speech perception using a compelling audiovisual adaptor. , 1994, The Journal of the Acoustical Society of America.

[20]  Arthur G. Samuel,et al.  Phonemic Restoration: Insights From a New Methodology , 1981 .

[21]  Q Summerfield,et al.  Audiovisual presentation demonstrates that selective adaptation in speech perception is purely auditory , 1981, Perception & psychophysics.

[22]  Mikko Sams,et al.  Where and When are the Heard and Seen Speech Integrated: Magnetoencephalographical (MEG) Studies , 1996 .

[23]  J. O'neill.  Contributions of the visual components of oral symbols to speech comprehension. , 1954, The Journal of speech and hearing disorders.

[24]  K. Sekiyama,et al.  Cultural and linguistic factors in audiovisual speech processing: The McGurk effect in Chinese subjects , 1997, Perception & psychophysics.

[25]  K. Green.  The Use of Auditory and Visual Information in Phonetic Perception , 1996 .

[26]  Y. Tohkura,et al.  McGurk effect in non-English listeners: few visual effects for Japanese subjects hearing Japanese syllables of high auditory intelligibility. , 1991, The Journal of the Acoustical Society of America.

[27]  Barbara Dodd,et al.  The Role of Vision in the Perception of Speech , 1977, Perception.

[28]  David G. Stork,et al.  Speechreading by Humans and Machines , 1996 .

[29]  C. Fowler,et al.  Listening with eye and hand: Cross-modal contributions to speech perception. , 1991 .

[30]  R. Hari,et al.  Seeing speech: visual information from lip movements modifies activity in the human auditory cortex , 1991, Neuroscience Letters.

[31]  A. Samuel.  Lexical uniqueness effects on phonemic restoration , 1987 .

[32]  Lawrence Brancazio,et al.  Lexical influences on the McGurk effect , 1997, AVSP.

[33]  H. McGurk,et al.  Visual influences on speech perception processes , 1978, Perception & psychophysics.