Developmental factors and the non-native speaker effect in auditory-visual speech perception.

The study investigated language and developmental factors in the use of visual information in audiovisual speech perception in speakers from different language backgrounds. Mandarin-Chinese and English adults and 8- to 9-year-old children were presented with /ba/, /da/, and /ga/ tokens spoken by two English and two Mandarin-Chinese speakers. A syllable identification task was presented in auditory-only, visual-only, and audiovisual (congruent and incongruent) conditions in clear and in noise. The results showed an increase in the use of visual information in adults relative to children in both the Chinese and English groups. In addition, a positive correlation between the total visual effect and speechreading performance was found, suggesting that the smaller visual influence in the bimodal condition for children might be accounted by their less sophisticated speechreading ability. In regard to the language factor, it was found that Chinese perceivers use visual information in their audiovisual speech processing to the same extent as English perceivers. Finally, there was evidence for a "non-native speaker effect" (i.e., stronger visual effect for non-native speech stimuli), but only for the English participants. Results from the current study suggest that the visual appearance of individual speakers and the acoustic-phonetic properties of specific languages should be considered in future cross-language studies.

[1]  D. Reisberg,et al.  Easy to hear but hard to understand: A lip-reading advantage with intact auditory stimuli. , 1987 .

[2]  Toshiko Isei-Jaakkola,et al.  Cognition and Physio-acoustic Correlates - Audio and Audio-visual Effects of a Short English Emotional Statement: On JL2, FL2 and EL1 , 2006, FinTAL.

[3]  L A Thompson,et al.  Developmental changes in visual and auditory contributions to speech perception. , 1986, Journal of experimental child psychology.

[4]  Linda Polka,et al.  A developmental study of audiovisual speech perception using the McGurk paradigm , 1994 .

[5]  P. Hoole,et al.  Tone-Vowel Interaction in Standard Chinese , 2004 .

[6]  Jacques de Villiers,et al.  New tools for interactive speech and language training: Using animated conversational agents in the classrooms of profoundly deaf children , 1999 .

[7]  Jean Vroomen,et al.  Auditory and visual speech perception in alphabetic and non-alphabetic Chinese-Dutch bilinguals , 1992 .

[8]  Denis Burnham,et al.  Auditory-visual speech perception development in Japanese and English speakers , 2003, AVSP.

[9]  Trevor H. Chen,et al.  Mandarin speech perception by ear and eye follows a universal principle , 2004, Perception & psychophysics.

[10]  Valérie Hazan,et al.  The effect of auditory and visual degradation on audiovisual perception of native and non-native speakers , 2008, INTERSPEECH.

[11]  Donna Erickson,et al.  Effect of Tone Height on Jaw and Tongue Articulation in Mandarin Chinese , 2004 .

[12]  K. Sekiyama,et al.  Cultural and linguistic factors in audiovisual speech processing: The McGurk effect in Chinese subjects , 1997, Perception & psychophysics.

[13]  Dominic W. Massaro,et al.  Cross-linguistic comparisons in the integration of visual and auditory speech , 1995, Memory & cognition.

[14]  D W Massaro,et al.  Children's perception of visual and auditory speech. , 1984, Child development.

[15]  Yue Wang,et al.  Effect of native language experience on audio-visual perception of English fricatives by Korean and Mandarin natives , 2007, AVSP.

[16]  W. H. Sumby,et al.  Visual contribution to speech intelligibility in noise , 1954 .

[17]  Trevor H. Chen,et al.  Seeing pitch: visual information for lexical tones of Mandarin-Chinese. , 2008, The Journal of the Acoustical Society of America.

[18]  Antoinette T. Gesi,et al.  Bimodal speech perception: an examination across languages , 1993 .

[19]  Angela Fuster-Duran.  Perception of Conflicting Audio-Visual Speech: an Examination across Spanish and German , 1996 .

[20]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[21]  Kaoru Sekiyama,et al.  Native-Foreign Langage Effect In The McGurk Effect : A Test With Chinese and Japanese , 1998, AVSP.

[22]  D. Burnham,et al.  Impact of language on development of auditory-visual speech perception. , 2008, Developmental science.

[23]  Y. Tohkura,et al.  Inter-language differences in the influence of visual cues in speech perception. , 1993 .