Asking a naive question about the McGurk effect: Why does audio [b] give more [d] percepts with visual [g] than with visual [d]?

Why does audio [b] give more [d] percepts with visual [g] than with visual [d], as in the present classical McGurk experiment ? An explanation given for this asymmetry could be a language bias towards [d]. As concerns the visual information, and contrary to what is sometimes taken as granted in part of the lipreading literature, visual [g] does not give more [d] than [g] responses. In fact [d] and [g] are neither visemically, nor auditorily equivalent. They are fully distinguishable in audio as well as in vision, where 80% correct identifications are current in laboratory condition, as in the present experiment. We show here that in spite of these highly differenciating scores, FLMP modelling can quite surprisingly account for such an asymmetry by tuning very small remaining values; which is highly unsatisfactory. We suggest another explanation for this asymmetry which could be grounded on brain mechanisms dedicated to the computation of auditory and visual mouth opening movements, i.e. audio movement and visual velocity detectors and integrators, dedicated to the bimodal integration of place targets in speech.

[1]  P. Gribble,et al.  Temporal constraints on the McGurk effect , 1996, Perception & psychophysics.

[2]  N. P. Erber,et al.  Auditory, visual, and auditory-visual recognition of consonants by children with normal and impaired hearing. , 1972, Journal of speech and hearing research.

[3]  A. Liberman,et al.  Some effects of later-occurring information on the perception of stop consonant and semivowel , 1979, Perception & psychophysics.

[4]  Y. Tohkura,et al.  McGurk effect in non-English listeners: few visual effects for Japanese subjects hearing Japanese syllables of high auditory intelligibility. , 1991, The Journal of the Acoustical Society of America.

[5]  C. G. Fisher,et al.  Confusions among visually perceived consonants. , 1968, Journal of speech and hearing research.

[6]  R. Campbell,et al.  Hearing by eye 2 : advances in the psychology of speechreading and auditory-visual speech , 1997 .

[7]  Ken W Grant,et al.  Hearing by Eye II: Advances in the Psychology of Speechreading and Auditory–Visual Speech, edited by Ruth Campbell, Barbara Dodd, and Denis Burnham , 1999, Trends in Cognitive Sciences.

[8]  A A Montgomery,et al.  Auditory and visual contributions to the perception of consonants. , 1974, Journal of speech and hearing research.

[9]  R. Campbell,et al.  Hearing by eye : the psychology of lip-reading , 1988 .

[10]  Dominic W. Massaro,et al.  Illusions and Issues In Bimodal Speech Perception , 1998, AVSP.

[11]  D. Massaro Speech Perception By Ear and Eye: A Paradigm for Psychological Inquiry , 1989 .

[12]  Monique Radeau,et al.  Etude comportementale et électrophysiologique des processus impliqués dans l'effet Mcgurk et dans l'effet de ventriloquie , 2001 .

[13]  S. Ohman Coarticulation in VCV utterances: spectrographic measurements. , 1966, The Journal of the Acoustical Society of America.

[14]  Philip K. McGuire,et al.  Activation in auditory cortex by speechreading in hearing people: FMRI studies , 1999, AVSP ...

[15]  K G Munhall,et al.  Audiovisual gating and the time course of speech perception. , 1998, The Journal of the Acoustical Society of America.

[16]  B. Walden,et al.  Effects of training on the visual recognition of consonants. , 1977, Journal of speech and hearing research.

[17]  G. A. Miller,et al.  An Analysis of Perceptual Confusions Among Some English Consonants , 1955 .

[18]  J. L. Miller,et al.  On the role of visual rate information in phonetic perception , 1985, Perception & psychophysics.

[19]  M. D. Wang,et al.  Consonant confusions in noise: a study of perceptual features. , 1973, The Journal of the Acoustical Society of America.

[20]  Q. Summerfield Some preliminaries to a comprehensive account of audio-visual speech perception. , 1987 .

[21]  E. DeYoe,et al.  A comparison of visual and auditory motion processing in human cerebral cortex. , 2000, Cerebral cortex.

[22]  Q Summerfield,et al.  Use of Visual Information for Phonetic Perception , 1979, Phonetica.

[23]  H. McGurk,et al.  Visual influences on speech perception processes , 1978, Perception & psychophysics.

[24]  H. McGurk,et al.  La langue et les Lèvres: cross-language influences on bimodal speech perception. , 1992, Canadian journal of psychology.

[25]  E. Owens,et al.  Visemes observed by hearing-impaired and normal-hearing adult viewers. , 1985, Journal of speech and hearing research.

[26]  R. Fox Modularity and the Motor Theory of Speech Perception , 1994 .