Beyond emotion archetypes: Databases for emotion modelling using neural networks

There has been rapid development in conceptions of the kind of database that is needed for emotion research. Familiar archetypes are still influential, but the state of the art has moved beyond them. There is concern to capture emotion as it occurs in action and interaction ('pervasive emotion') as well as in short episodes dominated by emotion, and therefore in a range of contexts, which shape the way it is expressed. Context links to modality-different contexts favour different modalities. The strategy of using acted data is not suited to those aims, and has been supplemented by work on both fully natural emotion and emotion induced by various technique that allow more controlled records. Applications for that kind of work go far beyond the 'trouble shooting' that has been the focus for application: 'really natural language processing' is a key goal. The descriptions included in such a database ideally cover quality, emotional content, emotion-related signals and signs, and context. Several schemes are emerging as candidates for describing pervasive emotion. The major contemporary databases are listed, emphasising those which are naturalistic or induced, multimodal, and influential.

[1]  Kostas Karpouzis,et al.  Emotion Analysis in Man-Machine Interaction Systems , 2004, MLMI.

[2]  P. Roach,et al.  Transcription of Prosodic and Paralinguistic Features of Emotional Speech , 1998, Journal of the International Phonetic Association.

[3]  Roddy Cowie,et al.  Emotional speech: Towards a new generation of databases , 2003, Speech Commun..

[4]  Nigel Ward,et al.  On the Expressive Competencies Needed for Responsive Systems , 2003 .

[5]  W. Schneider.  VAM: A neuro-cognitive model for visual attention control of segmentation, object recognition, and space-based motor action , 1995 .

[6]  David T. Lykken.  Polygraphic interrogation , 1984, Nature.

[7]  Miriam Kienast,et al.  Acoustical analysis of spectral and temporal changes in emotional speech , 2000 .

[8]  Multimodal data in action and interaction : a library of recordings and labelling schemes , 2004 .

[9]  J. Cacioppo,et al.  The psychophysiology of emotion. , 1993 .

[10]  Steven J. Simske,et al.  Recognition of emotions in interactive voice response systems , 2003, INTERSPEECH.

[11]  Piotr S. Szczepaniak,et al.  Computational intelligence and applications , 1999 .

[12]  Randolph R. Cornelius,et al.  The science of emotion: Research and tradition in the psychology of emotion. , 1997 .

[13]  P. Roach,et al.  TECHNIQUES FOR THE PHONETIC DESCRIPTION OF EMOTIONAL SPEECH , 2000 .

[14]  P. Ekman,et al.  The Repertoire of Nonverbal Behavior: Categories, Origins, Usage, and Coding , 1969 .

[15]  Andreas Stolcke,et al.  Prosody-based automatic detection of annoyance and frustration in human-computer dialog , 2002, INTERSPEECH.

[16]  V. Mountcastle,et al.  Higher functions of the brain , 1987 .

[17]  Klaus R. Scherer,et al.  Lost Luggage: A Field Study of Emotion–Antecedent Appraisal , 1997 .

[18]  C. Berger,et al.  Language and social knowledge: Uncertainty in interpersonal relations , 1982 .

[19]  Albert Rilliard,et al.  Why and how to control the authentic emotional speech corpora , 2003, INTERSPEECH.

[20]  Marc Schröder,et al.  Piecing Together the Emotion Jigsaw , 2004, MLMI.

[21]  E. Velten.  A laboratory task for induction of mood states. , 1968, Behaviour research and therapy.

[22]  McGilloway Ms.  Negative symptoms and speech parameters in schizophrenia. , 1997 .

[23]  A. Ortony,et al.  What's basic about basic emotions? , 1990, Psychological review.

[24]  D. Signorini,et al.  Neural networks , 1995, The Lancet.

[25]  Roddy Cowie,et al.  Emotion Recognition and Synthesis Based on MPEG‐4 FAPs , 2002 .

[26]  I. Iriondo,et al.  VALIDATION OF AN ACOUSTICAL MODELLING OF EMOTIONAL EXPRESSION IN SPANISH USING SPEECH SYNTHESIS TECHNIQUES , 2000 .

[27]  Pamela Kenealy,et al.  Validation of a music mood induction procedure: Some preliminary findings , 1988 .

[28]  Marc Schröder,et al.  Experimental study of affect bursts , 2003, Speech Commun..

[29]  Jennifer Healey,et al.  Toward Machine Emotional Intelligence: Analysis of Affective Physiological State , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[30]  R. Cowie,et al.  What are people doing when they assign everyday emotion terms , 2005 .

[31]  L. F. Barrett,et al.  Handbook of Emotions , 1993 .

[32]  R. Descartes,et al.  The Philosophical Works of Descartes , 1967 .

[33]  K. Scherer,et al.  Acoustic profiles in vocal emotion expression. , 1996, Journal of personality and social psychology.

[34]  P. Carruthers,et al.  The Philosophy of Psychology: Consciousness: the final frontier? , 1999 .

[35]  Ryohei Nakatsu,et al.  Emotion recognition and its application to computer agents with spontaneous interactive capabilities , 1999, Proceedings IEEE International Conference on Multimedia Computing and Systems.

[36]  Nick Campbell,et al.  Recording and storing of speech data , 2002 .

[37]  P. Lang.  International Affective Picture System (IAPS) : Technical Manual and Affective Ratings , 1995 .

[38]  P. Ekman.  Pictures of Facial Affect , 1976 .

[39]  T. Dalgleish.  Basic Emotions , 2004 .

[40]  J. Stainer,et al.  The Emotions , 1922, Nature.

[41]  Nigel Ward,et al.  Responding to subtle, fleeting changes in the user's internal state , 2001, CHI.

[42]  P. Ekman,et al.  Unmasking the face : a guide to recognizing emotions from facial clues , 1975 .

[43]  Roddy Cowie,et al.  What a neural net needs to know about emotion words , 1999 .

[44]  N. Campbell.  What type of inputs will we need for expressive speech synthesis? , 2002, Proceedings of 2002 IEEE Workshop on Speech Synthesis, 2002..

[45]  F. Pollick,et al.  Recognising Facial Expression from Spatially and Temporally Modified Movements , 2003, Perception.

[46]  M. Cabanac.  What is emotion? , 2002, Behavioural Processes.

[47]  Nick Campbell.  Specifying Affect and Emotion for Expressive Speech Synthesis , 2004, CICLing.

[48]  I. Linnankoski,et al.  Expression or emotional-motivational connotations with a one-word utterance. , 1997, The Journal of the Acoustical Society of America.

[49]  Roddy Cowie,et al.  Automatic recognition of emotion from voice: a rough benchmark , 2000 .

[50]  John Sabini,et al.  TARGET ARTICLE: Why Emotion Names and Experiences Don't Neatly Pair , 2005 .

[51]  K. Scherer,et al.  Effect of experimentally induced stress on vocal parameters. , 1986, Journal of experimental psychology. Human perception and performance.

[52]  J. Russell,et al.  Core affect, prototypical emotional episodes, and other things called emotion: dissecting the elephant. , 1999, Journal of personality and social psychology.

[53]  K. Scherer,et al.  Criteria for Emotion Recognition from Verbal and Nonverbal Expression: Studying Baggage Loss in the Airport , 2000 .

[54]  Elmar Nöth,et al.  “You Stupid Tin Box” - Children Interacting with the AIBO Robot: A Cross-linguistic Emotional Speech Corpus , 2004, LREC.

[55]  E. Rosch,et al.  Relationships among goodness-of-example, category norms, and word frequency , 1976 .

[56]  P. Laukka,et al.  Communication of emotions in vocal expression and music performance: different channels, same code? , 2003, Psychological bulletin.

[57]  Albino Nogueiras,et al.  Speech emotion recognition using hidden Markov models , 2001, INTERSPEECH.

[58]  Sjl Mozziconacci.  Speech variability and emotion : production and perception , 1998 .

[59]  J. Bachorowski.  Vocal Expression and Perception of Emotion , 1999 .

[60]  David M. Clark,et al.  On the induction of depressed mood in the laboratory: Evaluation and comparison of the velten and musical procedures , 1983 .

[61]  Roddy Cowie,et al.  Describing the emotional states that are expressed in speech , 2003, Speech Commun..

[62]  N. Amir,et al.  Analysis of an emotional speech corpus in Hebrew based on objective criteria , 2000 .

[63]  Alex Waibel,et al.  EMOTION-SENSITIVE HUMAN-COMPUTER INTERFACES , 2000 .

[64]  D. Mitchell Wilkes,et al.  Acoustical properties of speech as indicators of depression and suicidal risk , 2000, IEEE Transactions on Biomedical Engineering.

[65]  J. Fodor,et al.  Folk Psychology from the Standpoint of Conceptual Analysis , 1996 .

[66]  R. Descartes,et al.  The passions of the soul , 1989 .

[67]  P. Greasley,et al.  Emotion in Language and Speech: Methodological Issues in Naturalistic Approaches , 2000, Language and speech.

[68]  Soo-Jin Chung.  L'expression et la perception de l'émotion extraite de la parole spontanée : évidences du coréen et de l'anglais , 2000 .

[69]  Shrikanth S. Narayanan,et al.  Emotion recognition using a data-driven fuzzy inference system , 2003, INTERSPEECH.

[70]  W. James.  II.—WHAT IS AN EMOTION ? , 1884 .

[71]  P. Ekman,et al.  DIFFERENCES Universals and Cultural Differences in the Judgments of Facial Expressions of Emotion , 2004 .

[72]  Laurence Devillers,et al.  Reliability of Lexical and Prosodic Cues in Two Real-life Spoken Dialog Corpora , 2004, LREC.

[73]  Roddy Cowie,et al.  Speech in the Process of Becoming Bored , 2003 .

[74]  T. Dalgleish,et al.  Handbook of cognition and emotion , 1999 .

[75]  Petri Laukka.  Vocal Expression of Emotion Discrete-emotions and Dimensional Accounts , 2004 .

[76]  Richard J. Davidson,et al.  Neuropsychological Perspectives on Affective Styles and Their Cognitive Consequences , 2005 .

[77]  Elmar Nöth,et al.  How to find trouble in communication , 2003, Speech Commun..

[78]  Algirdas Pakstas,et al.  MPEG-4 Facial Animation: The Standard,Implementation and Applications , 2002 .

[79]  Rosalind W. Picard,et al.  Modeling drivers' speech under stress , 2003, Speech Commun..

[80]  Shrikanth S. Narayanan,et al.  Toward detecting emotions in spoken dialogs , 2005, IEEE Transactions on Speech and Audio Processing.

[81]  J. Gross,et al.  Emotion elicitation using films , 1995 .