Measures of auditory-visual integration in nonsense syllables and sentences.

For all but the most profoundly hearing-impaired (HI) individuals, auditory-visual (AV) speech has been shown consistently to afford more accurate recognition than auditory (A) or visual (V) speech. However, the amount of AV benefit achieved (i.e., the superiority of AV performance in relation to unimodal performance) can differ widely across HI individuals. To begin to explain these individual differences, several factors need to be considered. The most obvious of these are deficient A and V speech recognition skills. However, large differences in individuals' AV recognition scores persist even when unimodal skill levels are taken into account. These remaining differences might be attributable to differing efficiency in the operation of a perceptual process that integrates A and V speech information. There is at present no accepted measure of the putative integration process. In this study, several possible integration measures are compared using both congruent and discrepant AV nonsense syllable and sentence recognition tasks. Correlations were tested among the integration measures, and between each integration measure and independent measures of AV benefit for nonsense syllables and sentences in noise. Integration measures derived from tests using nonsense syllables were significantly correlated with each other; on these measures, HI subjects show generally high levels of integration ability. Integration measures derived from sentence recognition tests were also significantly correlated with each other, but were not significantly correlated with the measures derived from nonsense syllable tests. Similarly, the measures of AV benefit based on nonsense syllable recognition tests were found not to be significantly correlated with the benefit measures based on tests involving sentence materials. Finally, there were significant correlations between AV integration and benefit measures derived from the same class of speech materials, but nonsignificant correlations between integration and benefit measures derived from different classes of materials. These results suggest that the perceptual processes underlying AV benefit and the integration of A and V speech information might not operate in the same way on nonsense syllable and sentence input.

[1]  W. H. Sumby,et al.  Visual contribution to speech intelligibility in noise , 1954 .

[2]  G. A. Miller,et al.  An Analysis of Perceptual Confusions Among Some English Consonants , 1955 .

[3]  D. Massaro.  Preperceptual images, processing time, and perceptual units in auditory perception. , 1972, Psychological review.

[4]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[5]  B E Walden,et al.  Some effects of training on speech recognition by hearing-impaired adults. , 1981, Journal of speech and hearing research.

[6]  Q. Summerfield,et al.  Intermodal timing relations and audio-visual speech recognition by normal-hearing adults. , 1985, The Journal of the Acoustical Society of America.

[7]  Louis D. Braida,et al.  Evaluating the articulation index for auditory-visual input. , 1987, The Journal of the Acoustical Society of America.

[8]  A. Boothroyd,et al.  Mathematical treatment of context effects in phoneme and word recognition. , 1988, The Journal of the Acoustical Society of America.

[9]  B E Walden,et al.  Visual biasing of normal and impaired auditory speech perception. , 1990, Journal of speech and hearing research.

[10]  L. Braida.  Crossmodal Integration in the Identification of Consonant Segments , 1991, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[11]  A. Meltzoff,et al.  Integrating speech information across talkers, gender, and sensory modality: Female faces and male voices in the McGurk effect , 1991, Perception & psychophysics.

[12]  Marjorie R. Leek,et al.  Informational masking and auditory attention , 1991, Perception & psychophysics.

[13]  Q. Summerfield,et al.  Lipreading and audio-visual speech perception. , 1992, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[14]  W M Rabinowitz,et al.  Relations among different measures of speech reception in subjects using a cochlear implant. , 1992, The Journal of the Acoustical Society of America.

[15]  I. Winkler,et al.  Event-related brain potentials reflect traces of echoic memory in humans , 1993, Perception & psychophysics.

[16]  Ken W. Grant,et al.  Evaluating the articulation index for auditory–visual consonant recognition , 1993 .

[17]  D W Massaro,et al.  Perception of asynchronous and conflicting visual and auditory speech. , 1996, The Journal of the Acoustical Society of America.

[18]  L. Bernstein,et al.  Generalizability of speechreading performance on nonsense syllables, words, and sentences: subjects with normal hearing. , 1996, Journal of speech and hearing research.

[19]  C. Watson,et al.  Auditory and visual speech perception: confirmation of a modality-independent source of individual differences in speech recognition. , 1996, The Journal of the Acoustical Society of America.

[20]  A Baddeley,et al.  The fractionation of working memory. , 1996, Proceedings of the National Academy of Sciences of the United States of America.

[21]  W O Olsen,et al.  Phoneme and Word Recognition for Words in Isolation and in Sentences , 1997, Ear and hearing.

[22]  K. Grant,et al.  Auditory-visual speech recognition by hearing-impaired subjects: consonant recognition, sentence recognition, and auditory-visual integration. , 1998, The Journal of the Acoustical Society of America.