Interaction of Audition and Vision for the Perception of Prosodic Contrastive Focus

Prosodic contrastive focus is used to attract the listener's attention to a specific part of the utterance. Mostly conceived of as auditory/acoustic, it also has visible correlates which have been shown to be perceived. This study aimed at analyzing auditory-visual perception of prosodic focus by elaborating a paradigm enabling an auditory-visual advantage measurement (avoiding the ceiling effect) and by examining the interaction between audition and vision. A first experiment proved the efficiency of a whispered speech paradigm to measure an auditory-visual advantage for the perception of prosodic features. A second experiment used this paradigm to examine and characterize the auditory-visual perceptual processes. It combined performance assessment (focus detection score) to reaction time measurements and confirmed and extended the results from the first experiment. This study showed that adding vision to audition for perception of prosodic focus can not only improve focus detection but also reduce reaction times. A further analysis suggested that audition and vision are actually integrated for the perception of prosodic focus. Visual-only perception appeared to be facilitated for whispered speech suggesting an enhancement of visual cues in whispering. Moreover, the potential influence of the presence of facial markers on perception is discussed.

[1]  D. Reisberg,et al.  Easy to hear but hard to understand: A lip-reading advantage with intact auditory stimuli. , 1987 .

[2]  Paul Touati,et al.  Structures prosodiques du suédois et du français : profils temporels et configurations tonales , 1987 .

[3]  Carlos Gussenhoven,et al.  Testing the Reality of Focus Domains , 1983 .

[4]  Jean-Luc Schwartz,et al.  Visual perception of contrastive focus in reiterant French speech , 2004, Speech Commun..

[5]  Mohamed Tahar Lallouache,et al.  Un poste "visage-parole" couleur : acquisition et traitement automatique des contours des lèvres , 1991 .

[6]  Björn Granström,et al.  Visual correlates to prominence in several expressive modes , 2006, INTERSPEECH.

[7]  Dorothy Mossman Thompson,et al.  On the Detection of Emphasis in Spoken Sentences by Means of Visual, Tactual, and Visual-Tactual Cues , 1934 .

[8]  D. Ingvar,et al.  Disturbances of speech prosody following right hemisphere infarcts , 1991, Acta neurologica Scandinavica.

[9]  Taehong Cho.  Prosodic strengthening and featural enhancement: evidence from acoustic and articulatory realizations of /a,i/ in English. , 2005, The Journal of the Acoustical Society of America.

[10]  Kenneth L. Pike.  On the Grammar of Intonation , 1965 .

[11]  A. Risberg,et al.  On the identification of intonation contours by hearing impaired listeners , 2007 .

[12]  Eric Vatikiotis-Bateson,et al.  Rhythm type and articulatory dynamics in English, French and Japanese , 1993 .

[13]  L. Bernstein,et al.  Single-channel vibrotactile supplements to visual perception of intonation and stress. , 1989, The Journal of the Acoustical Society of America.

[14]  AN INVESTIGATION OF ARTICULATORY CORRELATES OF THE ACCENTUAL PHRASE IN FRENCH , 1999 .

[15]  Marilyn M. Vihman,et al.  Phonological Development , 2014 .

[16]  G. Bailly,et al.  Virtual talking heads and ambiant face-to- face communication , 2006 .

[17]  A. Macleod,et al.  Quantifying the contribution of vision to speech perception in noise. , 1987, British journal of audiology.

[18]  Volker Strom,et al.  Visual prosody: facial movements accompanying speech , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[19]  Emiel Krahmer,et al.  Facial expression and prosodic prominence: Effects of modality and facial area , 2008, J. Phonetics.

[20]  K. D. de Jong.  The supraglottal articulation of prominence in English: linguistic stress as localized hyperarticulation. , 1995, The Journal of the Acoustical Society of America.

[21]  Jeffery A. Jones,et al.  Visual Prosody and Speech Intelligibility , 2004, Psychological science.

[22]  E H Buder,et al.  Acoustic correlates of stress in young children's speech. , 1995, Journal of speech and hearing research.

[23]  P. Keating,et al.  Optical Phonetics and Visual Perception of Lexical and Phrasal Stress in English , 2009, Language and speech.

[24]  Elisabeth Selkirk,et al.  Phonology and Syntax: The Relation between Sound and Structure , 1984 .

[25]  J. Beskow,et al.  Visual correlates to prominence in , 2006 .

[26]  Björn Granström,et al.  Audiovisual representation of prosody in expressive speech communication , 2004, Speech Commun..

[27]  W. H. Sumby,et al.  Visual contribution to speech intelligibility in noise , 1954 .

[28]  Donna Erickson,et al.  Articulation of Extreme Formant Patterns for Emphasized Vowels , 2002, Phonetica.

[29]  Sharon Peperkamp,et al.  Discovering words in the continuous speech stream: the role of prosody , 2003, J. Phonetics.

[30]  M. Ouellet,et al.  L'INTONATION, LE SYSTÈME DU FRANÇAIS : DESCRIPTION ET MODÉLISATION , 2000 .

[31]  D. Crystal,et al.  Intonation and Grammar in British English , 1967 .

[32]  Mariapaola D'Imperio,et al.  Focus and tonal structure in Neapolitan Italian , 2001, Speech Commun..

[33]  M. Mesulam,et al.  Disturbances in prosody. A right-hemisphere contribution to language. , 1981, Archives of neurology.

[34]  Emiel Krahmer,et al.  Pitch, eyebrows and the perception of focus , 2002, Speech Prosody 2002.

[35]  G. A. Miller,et al.  An Analysis of Perceptual Confusions Among Some English Consonants , 1955 .

[36]  Karen Bryan,et al.  Language prosody and the right hemisphere , 1989 .

[37]  K. D. Jong.  The supraglottal articulation of prominence in English: Linguistic stress as localized hyperarticulation , 1995 .

[38]  H. Hill,et al.  Visual Correlates of Prosodic Contrastive Focus in French: Description and Inter-Speaker Variability , 2006 .

[39]  C. Clifton,et al.  Focus, Accent, and Argument Structure: Effects on Language Comprehension , 1995, Language and speech.

[40]  J. Harrington,et al.  Coarticulation and the accented/unaccented distinction: evidence from jaw movement data , 1995 .

[41]  H. Nølke.  Linguistique modulaire: de la forme au sens , 1994 .

[42]  J. Kelso,et al.  A qualitative dynamic analysis of reiterant speech production: phase portraits, kinematics, and dynamic modeling. , 1985, The Journal of the Acoustical Society of America.

[43]  Jianwu Dang,et al.  Some articulatory and acoustic changes associated with emphasis in spoken English , 2000, INTERSPEECH.

[44]  L. Danon-Boileau,et al.  Grammaire de l'intonation : l'exemple du français , 1998 .

[45]  F. Minifie,et al.  Acoustical-perceptual correlates of "whisper pitch" in synthetically generated vowels. , 1999, Journal of speech, language, and hearing research : JSLHR.

[46]  M. Swerts,et al.  Congruent and incongruent audiovisual cues to prominence , 2004, Speech Prosody 2004.

[47]  Hélène Loevenbruck.  Pistes pour le contrôle d'un robot parlant capable de réduction vocalique , 1996 .

[48]  U. Hadar,et al.  Head Movement Correlates of Juncture and Stress at Sentence Level , 1983, Language and speech.

[49]  Shari R. Baum,et al.  Sentence comprehension by Broca's aphasics: Effects of some suprasegmental variables , 1982, Brain and Language.

[50]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[51]  Roxane Bertrand,et al.  About the relationship between eyebrow movements and Fo variations , 1996, Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96.

[52]  Emiel Krahmer,et al.  Perceptual evaluation of audiovisual cues for prominence , 2002, INTERSPEECH.

[53]  Michael Halliday,et al.  Intonation and Grammar in British English , 1967 .

[54]  D. Dahan,et al.  Interspeaker Variability in Emphatic Accent Production in French , 1996, Language and speech.

[55]  John Kingston,et al.  Salient pitch cues in the perception of contrastive focus , 1994 .

[56]  Marion Dohen,et al.  Audiovisual production and perception of contrastive focus in French: a multispeaker study , 2005, INTERSPEECH.

[57]  W. V. Summers.  Effects of stress and final-consonant voicing on vowel production: articulatory and acoustic analyses. , 1987, The Journal of the Acoustical Society of America.