"Your Tone Says It All": The processing and interpretation of affective language

Affective speech production, perception and action tasks were conducted.Talkers produce both systematic similarities and individual variations of acoustic cues in affective speech.Listeners accurately categorize this affective speech between and within talkers.Confidence in responding to affective speech is mediated by saliency of acoustic cues. Pragmatic interpretation of intent is essential for successful communication. The current studies evaluate the impact of affective prosody on the processing and interpretation of affectively spoken language. A production study provided further evidence of talker variability in the production of the emotionally-laden categories of Innuendo, Irritation, Compassion and Neutral, indicating a great deal of within and between talker variability, as well as talker systematicity within affect categories. Despite this talker variability, in a listening task, participants were asked to categorize the intent of the talkers statements (from the production study) to determine the relative accuracy of responding, while also tracking the perception of intent as it unfolded over time (i.e., via computer Wii-mote x, y coordinates). The results from the online measurement of the perception of intent indicated that even though our listeners were accurate in categorizing intent (~70% mean accuracy), the "dynamic signature" of their responses was laden with a great deal hesitation and indecision for some, but not all talkers. This suggests that during the perception of intent, the cognitive system is flexible enough to handle talker variability, but during perception, uncertainty will change the manner in which the intent is processed.

[1]  P. Rockwell,et al.  Lower, Slower, Louder: Vocal Cues of Sarcasm , 2000 .

[2]  Edward Gibson,et al.  Using Mechanical Turk to Obtain and Analyze English Acceptability Judgments , 2011, Lang. Linguistics Compass.

[3]  Sue Peppé,et al.  Intonation features of the expression of emotions in Spanish: preliminary study for a prosody assessment procedure , 2008, Clinical linguistics & phonetics.

[4]  D. Massaro,et al.  Fuzzy logical model of bimodal emotion perception: Comment on “The perception of emotions by ear and by eye” by de Gelder and Vroomen , 2000 .

[5]  P. Lang International affective picture system (IAPS) : affective ratings of pictures and instruction manual , 2005 .

[6]  Michael J. Spivey,et al.  The Continuity Of Mind , 2008 .

[7]  A. Manstead,et al.  Social functions of emotion , 2008 .

[8]  Jennifer M. Roche,et al.  Exploring Action Dynamics as an Index of Paired-Associate Learning , 2008, PloS one.

[9]  K. Scherer,et al.  Cue utilization in emotion attribution from auditory stimuli , 1977 .

[10]  Kim E. A. Silverman,et al.  Vocal cues to speaker affect: testing two models , 1984 .

[11]  Skyler T. Hawk,et al.  "Worth a thousand words": absolute and relative decoding of nonlinguistic affect vocalizations. , 2009, Emotion.

[12]  Paula M. Niedenthal,et al.  Resolution of Lexical Ambiguity by Emotional State , 1995 .

[13]  R. G. Crowder,et al.  Precategorical acoustic storage (PAS) , 1969 .

[14]  D. Pisoni,et al.  Talker-specific learning in speech perception , 1998, Perception & psychophysics.

[15]  C. D. De Dreu,et al.  The interpersonal effects of anger and happiness in negotiations. , 2004, Journal of personality and social psychology.

[16]  Julia Hirschberg,et al.  Classifying subject ratings of emotional speech using acoustic features , 2003, INTERSPEECH.

[17]  Gordon E. Legge,et al.  Learning unfamiliar voices , 1984 .

[18]  J. Vroomen,et al.  The perception of emotions by ear and by eye , 2000 .

[19]  Marc D. Pell,et al.  The sound of sarcasm , 2008, Speech Commun..

[20]  Henry C. Ellis,et al.  A convenient self-referencing mood induction procedure , 1991 .

[21]  Charles M. Galloway Nonverbal: The Language of Sensitivity. , 1971 .

[22]  Ruben C Gur,et al.  Validation of affective and neutral sentence content for prosodic testing , 2008, Behavior research methods.

[23]  Brendan T. O'Connor,et al.  Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks , 2008, EMNLP.

[24]  S. Kitayama,et al.  Spontaneous Attention to Word Content Versus Emotional Tone , 2003, Psychological science.

[25]  F. Colavita Human sensory dominance , 1974 .

[26]  Klaus R. Scherer,et al.  Vocal communication of emotion: A review of research paradigms , 2003, Speech Commun..

[27]  K. Scherer,et al.  Acoustic profiles in vocal emotion expression. , 1996, Journal of personality and social psychology.

[28]  T. Gilovich,et al.  The illusion of transparency: biased assessments of others' ability to read one's emotional states. , 1998, Journal of personality and social psychology.

[29]  Paul Boersma,et al.  Praat: doing phonetics by computer , 2003 .

[30]  D. Keltner,et al.  Social Functions of Emotions at Four Levels of Analysis , 1999 .

[31]  A. Majid Current Emotion Research in the Language Sciences , 2012 .

[32]  P. Ekman,et al.  Emotion in the Human Face: Guidelines for Research and an Integration of Findings , 1972 .

[33]  L. Nygaard,et al.  Communicating emotion: linking affective prosody and word meaning. , 2008, Journal of experimental psychology. Human perception and performance.

[34]  J. Bachorowski,et al.  Acoustic correlates of talker sex and individual talker identity are present in a short vowel segment produced in running speech. , 1999, The Journal of the Acoustical Society of America.

[35]  A. J. Fridlund Human Facial Expression: An Evolutionary View , 1994 .

[36]  P. Laukka,et al.  Impact of intended emotion intensity on cue utilization and decoding accuracy in vocal expression of emotion. , 2000, Emotion.

[37]  Richard J. Gerrig,et al.  How valence affects language processing: Negativity bias and mood congruence in narrative comprehension , 2009, Memory & cognition.

[38]  K. Scherer On the Symbolic Functions of Vocal Affect Expression , 1988 .

[39]  David B. Pisoni,et al.  Long-term memory in speech perception: Some new findings on talker variability, speaking rate and perceptual learning , 1993, Speech Commun..

[40]  Klaus R. Scherer,et al.  The role of intonation in emotional expressions , 2005, Speech Commun..

[41]  Tim Futing Liao,et al.  Multinomial Logit Models , 1994 .

[42]  S. Kitayama,et al.  Word and voice: Spontaneous attention to emotional utterances in two languages , 2002 .

[43]  I. Linnankoski,et al.  Expression or emotional-motivational connotations with a one-word utterance. , 1997, The Journal of the Acoustical Society of America.

[44]  Laura K. Guerrero,et al.  Nonverbal Communication in Close Relationships , 2005 .

[45]  A. Dijker Emotional reactions to ethnic minorities , 1987 .

[46]  Michael J. Spivey,et al.  Action Dynamics Reveal Parallel Competition in Decision Making , 2008, Psychological science.

[47]  David B. Pisoni,et al.  Learning to recognize talkers from natural, sinewave, and reversed speech samples. , 2002 .

[48]  R. Kraut,et al.  Social and emotional messages of smiling: An ethological approach. , 1979 .

[49]  K. Scherer,et al.  Introducing the Geneva Multimodal expression corpus for experimental research on emotion perception. , 2012, Emotion.

[50]  J. Richardson Precategorical acoustic storage and postcategorical lexical storage , 1979, Cognitive Psychology.

[51]  L. Nygaard,et al.  Resolution of lexical ambiguity by emotional tone of voice , 2002 .

[52]  K. Scherer Vocal affect expression: a review and a model for future research. , 1986, Psychological bulletin.

[53]  J. Mullennix,et al.  Effects of Variation in Emotional Tone of Voice on Speech Perception , 2002, Language and speech.

[54]  Jonathan B Freeman,et al.  MouseTracker: Software for studying real-time mental processing using a computer mouse-tracking method , 2010, Behavior research methods.

[55]  L. Kaiser Communication of affects by single vowels , 1962, Synthese.

[56]  Neil A. Macmillan,et al.  Resolution for speech sounds: basic sensitivity and context memory on vowel and consonant continua , 1988 .

[57]  A. B.,et al.  SPEECH COMMUNICATION , 2001 .

[58]  Howard C. Nusbaum,et al.  Emotional language processing: How mood affects integration processes during discourse comprehension , 2012, Brain and Language.

[59]  K. Scherer,et al.  Geneva Emotion Wheel Rating Study , 2012 .

[60]  D. H. Warren,et al.  Sensory conflict in judgments of spatial direction , 1969 .

[61]  Michael J. Spivey,et al.  Graded motor responses in the time course of categorizing atypical exemplars , 2007, Memory & cognition.

[62]  Kristen N. Moreno,et al.  How do I feel about them? The role of affective reactions in intergroup perception. , 2000 .

[63]  Klaus R. Scherer,et al.  Lost Luggage: A Field Study of Emotion–Antecedent Appraisal , 1997 .

[64]  J. Mullennix,et al.  Some effects of talker variability on spoken word recognition. , 1989, The Journal of the Acoustical Society of America.

[65]  S. Trehub,et al.  Children's understanding of emotion in speech. , 2001, Child development.

[66]  D H Brainard,et al.  The Psychophysics Toolbox. , 1997, Spatial vision.

[67]  I. Poggi,et al.  Multimodal markers of irony and sarcasm , 2003 .

[68]  Michael J. Spivey,et al.  Continuous attraction toward phonological competitors. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[69]  Paul Boersma,et al.  Praat, a system for doing phonetics by computer , 2002 .

[70]  G. Plant Perceiving Talking Faces: From Speech Perception to a Behavioral Principle , 1999 .

[71]  Per-Erik Nordström,et al.  Female and infant vocal tracts simulated from male area functions , 1977 .

[72]  J. Ryalls,et al.  A preliminary investigation of the effects of gender and race on Voice Onset Time. , 1997, Journal of speech, language, and hearing research : JSLHR.

[73]  M. T. Hynan The perinatal stress composite: A validation study , 1991 .

[74]  Michael J. Spivey,et al.  Tracking the Continuity of Language Comprehension: Computer Mouse Trajectories Suggest Parallel Syntactic Processing , 2007, Cogn. Sci..

[75]  K. Scherer,et al.  Emotion Inferences from Vocal Expression Correlate Across Languages and Cultures , 2001 .

[76]  J. Vroomen,et al.  Covert Processing of Faces in Prosopagnosia Is Restricted to Facial Expressions: Evidence from Cross-Modal Bias , 2000, Brain and Cognition.

[77]  Csr Young,et al.  How to Do Things With Words , 2009 .

[78]  D B Pisoni,et al.  Stimulus variability and processing dependencies in speech perception , 1990, Perception & psychophysics.

[79]  J. Bachorowski Vocal Expression and Perception of Emotion , 1999 .

[80]  P. Ekman Are there basic emotions? , 1992, Psychological review.

[81]  Jennifer M. Fellowes,et al.  Learning to recognize talkers from natural, sinewave, and reversed speech samples. , 2002, Journal of experimental psychology. Human perception and performance.

[82]  M. Swerts,et al.  Prosodic predictors of upcoming positive or negative content in spoken messages. , 2010, The Journal of the Acoustical Society of America.

[83]  R. Newman,et al.  The perceptual consequences of within-talker variability in fricative production. , 2001, The Journal of the Acoustical Society of America.

[84]  David A. Forsyth,et al.  Utility data annotation with Amazon Mechanical Turk , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[85]  Ursula Gisela Goldstein,et al.  An articulatory model for the vocal tracts of growing children , 1980 .

[86]  Annett Schirmer,et al.  On the role of attention for the processing of emotions in speech: sex differences revisited. , 2005, Brain research. Cognitive brain research.

[87]  T. Jaeger,et al.  Categorical Data Analysis: Away from ANOVAs (transformation or not) and towards Logit Mixed Models. , 2008, Journal of memory and language.

[88]  D. Pisoni,et al.  Speech Perception as a Talker-Contingent Process , 1993, Psychological science.

[89]  Thomas A. Farmer,et al.  Hand in Motion Reveals Mind in Motion , 2011, Front. Psychology.

[90]  J. Gibson,et al.  Adaptation, after-effect and contrast in the perception of curved lines. , 1933 .

[91]  Klaus R. Scherer,et al.  Using Actor Portrayals to Systematically Study Multimodal Emotion Expression: The GEMEP Corpus , 2007, ACII.

[92]  K. Hammerschmidt,et al.  Acoustical correlates of affective prosody. , 2007, Journal of voice : official journal of the Voice Foundation.

[93]  P. Laukka,et al.  Communication of emotions in vocal expression and music performance: different channels, same code? , 2003, Psychological bulletin.

[94]  M. Posner,et al.  Visual dominance: an information-processing account of its origins and significance. , 1976, Psychological review.

[95]  B. Mesquita,et al.  Cultural affordances and emotional experience: socially engaging and disengaging emotions in Japan and the United States. , 2006, Journal of personality and social psychology.

[96]  Sylvie J. L. Mozziconacci,et al.  Modeling Emotion and Attitude in Speech by Means of Perceptually Based Parameter Values , 2001, User Modeling and User-Adapted Interaction.

[97]  Dale T. Miller,et al.  Pluralistic ignorance: When similarity is interpreted as dissimilarity. , 1987 .