Segmental differences in the visual contribution to speech inteligibility

It is well known that the presence of visual cues increases the intelligibility of a speech signal (Sumby and Pollack, 1954). Although much is known about segmental differences in visual‐only perception, little is known about the contribution of visual cues to auditory–visual perception for individual segments. The purpose of this study was to examine (1) whether segments differ in their visual contribution to speech intelligibility, and (2) whether the contribution of visual cues is always to increase speech intelligibility. One talker produced triples of real words containing 15 different English consonants. Forced‐choice word‐identification experiments were carried out with these recordings under auditory–visual (AV) and auditory‐only (A) conditions with varying S/N ratios, and identification accuracy for the 15 consonants was compared for A versus AV conditions. As expected, there were significant differences in the visual contribution for the different consonants, with visual cues greatly improving s...

[1]  N. P. Erber.  Interaction of audition and vision in the recognition of oral speech stimuli. , 1969, Journal of speech and hearing research.

[2]  W. H. Sumby,et al.  Visual contribution to speech intelligibility in noise , 1954 .

[3]  A. Macleod,et al.  A procedure for measuring auditory and audio-visual speech-reception thresholds for sentences in noise: rationale, evaluation, and recommendations for use. , 1990, British journal of audiology.

[4]  Matthew Flatt,et al.  PsyScope: An interactive graphic system for designing and controlling experiments in the psychology laboratory using Macintosh computers , 1993 .

[5]  D. Massaro,et al.  Tests of auditory-visual integration efficiency within the framework of the fuzzy logical model of perception. , 2000, The Journal of the Acoustical Society of America.

[6]  P. Luce.  Neighborhoods of words in the mental lexicon , 1986 .

[7]  B. Walden,et al.  Effects of training on the visual recognition of consonants. , 1977, Journal of speech and hearing research.

[8]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[9]  D. Massaro,et al.  Perceiving Talking Faces , 1995 .

[10]  P K Kuhl,et al.  The role of visual information in the processing of , 1989, Perception & psychophysics.

[11]  Edward C. Chao,et al.  Generalized Estimating Equations , 2003, Technometrics.

[12]  Y. Tohkura,et al.  McGurk effect in non-English listeners: few visual effects for Japanese subjects hearing Japanese syllables of high auditory intelligibility. , 1991, The Journal of the Acoustical Society of America.

[13]  Jeng-Shiann Jiang.  Relating Optical Speech to Speech Acoustic and Visual Speech Perception , 2003 .

[14]  M. D. Wang,et al.  Consonant confusions in noise: a study of perceptual features. , 1973, The Journal of the Acoustical Society of America.

[15]  Q. Summerfield.  Some preliminaries to a comprehensive account of audio-visual speech perception. , 1987 .

[16]  G. A. Miller,et al.  Erratum: An Analysis of Perceptual Confusions Among Some English Consonants [J. Acoust. Soc. Am. 27, 339 (1955)] , 1955 .

[17]  A. King,et al.  Multisensory integration: perceptual grouping by eye and ear , 2001, Current Biology.

[18]  J. O'neill.  Contributions of the visual components of oral symbols to speech comprehension. , 1954, The Journal of speech and hearing disorders.

[19]  J. Pierrehumbert,et al.  Similarity and phonotactics in Arabic , 1997 .

[20]  S Rosen,et al.  A video-recorded test of lipreading for British English. , 1982, British journal of audiology.

[21]  P F Seitz,et al.  The use of visible speech cues for improving auditory detection of spoken sentences. , 2000, The Journal of the Acoustical Society of America.

[22]  A A Montgomery,et al.  Auditory and visual contributions to the perception of consonants. , 1974, Journal of speech and hearing research.

[23]  Y. Tohkura,et al.  Inter-language differences in the influence of visual cues in speech perception. , 1993 .

[24]  Antoinette T. Gesi,et al.  Bimodal speech perception: an examination across languages , 1993 .

[25]  N. P. Erber.  Auditory-visual perception of speech. , 1975, The Journal of speech and hearing disorders.

[26]  Abeer Alwan,et al.  Predicting visual consonant perception from physical measures , 2001, INTERSPEECH.

[27]  John J. Ohala.  Comparison of speech sounds: distance vs. cost metrics , 1997 .

[28]  Q Summerfield,et al.  Use of Visual Information for Phonetic Perception , 1979, Phonetica.

[29]  M. Woodward,et al.  Phoneme perception in lipreading. , 1960, Journal of speech and hearing research.

[30]  A. Macleod,et al.  Quantifying the contribution of vision to speech perception in noise. , 1987, British journal of audiology.