The Ordinal Nature of Emotions: An Emerging Approach

Computational representation of everyday emotional states is a challenging task and, arguably, one of the most fundamental for affective computing. Standard practice in emotion annotation is to ask people to assign a value of intensity or a class value to each emotional behavior they observe. Psychological theories and evidence from multiple disciplines including neuroscience, economics and artificial intelligence, however, suggest that the task of assigning reference-based values to subjective notions is better aligned with the underlying representations. This paper draws together the theoretical reasons to favor ordinal labels for representing and annotating emotion, reviewing the literature across several disciplines. We go on to discuss good and bad practices of treating ordinal and other forms of annotation data and make the case for preference learning methods as the appropriate approach for treating ordinal labels. We finally discuss the advantages of ordinal annotation with respect to both reliability and validity through a number of case studies in affective computing, and address common objections to the use of ordinal data. More broadly, the thesis that emotions are by nature ordinal is supported by both theoretical arguments and evidence, and opens new horizons for the way emotions are viewed, represented and analyzed computationally.

[1]  Touradj Ebrahimi,et al.  Subjective Quality Evaluation via Paired Comparison: Application to Scalable Video Coding , 2011, IEEE Transactions on Multimedia.

[2]  Roddy Cowie,et al.  Real life emotions in French and English TV video clips: an integrated annotation protocol combining continuous and discrete approaches , 2006, LREC.

[3]  H. B. Mann,et al.  On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other , 1947 .

[4]  Rainer Reisenzein,et al.  Metric Scales for Emotion Measurement , 2016 .

[5]  A. Tversky,et al.  Context-dependent preferences , 1993 .

[6]  Ioannis Patras,et al.  AMIGOS: A Dataset for Affect, Personality and Mood Research on Individuals and Groups , 2017, IEEE Transactions on Affective Computing.

[7]  M. Rokeach The Nature Of Human Values , 1974 .

[8]  Wolfgang G. Stock,et al.  Collective indexing of emotions in images. A study in emotional information retrieval , 2009, J. Assoc. Inf. Sci. Technol..

[9]  Andrea Bonarini,et al.  Modeling enjoyment preference from physiological responses in a car racing game , 2010, Proceedings of the 2010 IEEE Conference on Computational Intelligence and Games.

[10]  B. Mesquita,et al.  Context in Emotion Perception , 2011 .

[11]  Ingo Siegert,et al.  ikannotate - A Tool for Labelling, Transcription, and Annotation of Emotionally Coloured Speech , 2011, ACII.

[12]  Eliathamby Ambikairajah,et al.  An investigation of emotion change detection from speech , 2015, INTERSPEECH.

[13]  Eyke Hüllermeier,et al.  Preferences in AI: An overview , 2011, Artif. Intell..

[14]  Jon D. Morris Observations: SAM: The Self-Assessment Manikin An Efficient Cross-Cultural Measurement Of Emotional Response 1 , 1995 .

[15]  Björn W. Schuller,et al.  DeepCoder: Semi-Parametric Variational Autoencoders for Automatic Facial Action Coding , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[16]  W. Kruskal,et al.  Use of Ranks in One-Criterion Variance Analysis , 1952 .

[17]  Mohamed Chetouani,et al.  Multimodal Detection of Engagement in Groups of Children Using Rank Learning , 2016, HBU.

[18]  Yoshua Bengio,et al.  Learning deep physiological models of affect , 2013, IEEE Computational Intelligence Magazine.

[19]  Antonios Liapis,et al.  RankTrace: Relative and unbounded affect annotation , 2017, 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII).

[20]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[21]  R. Likert “Technique for the Measurement of Attitudes, A” , 2022, The SAGE Encyclopedia of Research Design.

[22]  M. Bradley,et al.  Measuring emotion: the Self-Assessment Manikin and the Semantic Differential. , 1994, Journal of behavior therapy and experimental psychiatry.

[23]  Carlos Busso,et al.  Preference-Learning with Qualitative Agreement for Sentence Level Emotional Annotations , 2018, INTERSPEECH.

[24]  Neil J. MacKinnon Measuring Self-Sentiments , 2015 .

[25]  Jing Li,et al.  Boosting paired comparison methodology in measuring visual discomfort of 3DTV: performances of three different designs , 2013, Electronic Imaging.

[26]  Jacob Beck,et al.  Adaptation-level theory. , 1974 .

[27]  Roddy Cowie,et al.  Induction, recording and recognition of natural emotions from facial expressions and speech prosody , 2013, Journal on Multimodal User Interfaces.

[28]  Carlos Busso,et al.  Using Agreement on Direction of Change to Build Rank-Based Emotion Classifiers , 2016, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[29]  Gregory N. Hullender,et al.  Learning to rank using gradient descent , 2005, ICML.

[30]  R. Nisbett,et al.  Validity problems comparing values across cultures and possible solutions. , 1997 .

[31]  Peter Goldie,et al.  Emotions, feelings and intentionality , 2002 .

[32]  S S Stevens,et al.  On the Theory of Scales of Measurement. , 1946, Science.

[33]  Georgios N. Yannakakis,et al.  To rank or to classify? Annotating stress for reliable PTSD profiling , 2015, 2015 International Conference on Affective Computing and Intelligent Interaction (ACII).

[34]  H. Helson Adaptation-level theory : an experimental and systematic approach to behavior , 1964 .

[35]  S. McIntyre,et al.  Developing Practical Procedures for the Measurement of Personal Values in Cross-Cultural Marketing , 1979 .

[36]  D. Kahneman A perspective on judgment and choice: mapping bounded rationality. , 2003, The American psychologist.

[37]  K. Scherer What are emotions? And how can they be measured? , 2005 .

[38]  R. Reisenzein,et al.  Indirect scaling methods for testing quantitative emotion theories , 2013, Cognition & emotion.

[39]  Ragini Verma,et al.  Speaker-sensitive emotion recognition via ranking: Studies on acted and spontaneous speech , 2015, Comput. Speech Lang..

[40]  R. Elliott,et al.  Medial orbitofrontal cortex codes relative rather than absolute value of financial rewards in humans , 2008, The European journal of neuroscience.

[41]  Louis-Philippe Morency,et al.  Relative facial action unit detection , 2014, IEEE Winter Conference on Applications of Computer Vision.

[42]  D. Heylen,et al.  Issues in Data Labelling , 2011 .

[43]  Touradj Ebrahimi,et al.  Paired comparison-based subjective quality assessment of stereoscopic images , 2013, Multimedia Tools and Applications.

[44]  G. Norman Likert scales, levels of measurement and the “laws” of statistics , 2010, Advances in health sciences education : theory and practice.

[45]  Grigorios Tsoumakas,et al.  Multi-Label Classification of Music into Emotions , 2008, ISMIR.

[46]  Ingo Siegert,et al.  The Influence of Context Knowledge for Multi-modal Affective Annotation , 2013, HCI.

[47]  Jon A. Krosnick,et al.  The Measurement of Values in Surveys: A Comparison of Ratings and Rankings , 1985 .

[48]  A. Tversky Features of Similarity , 1977 .

[49]  L. Cronbach Coefficient alpha and the internal structure of tests , 1951 .

[50]  Yi-Hsuan Yang,et al.  Music emotion ranking , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[51]  Eyke Hüllermeier,et al.  Preference Learning: An Introduction , 2010, Preference Learning.

[52]  Emily Mower Provost,et al.  Emotion spotting: discovering regions of evidence in audio-visual emotion expressions , 2016, ICMI.

[53]  Georgios N. Yannakakis,et al.  The Preference Learning Toolbox , 2015, ArXiv.

[54]  Shrikanth Narayanan,et al.  Toward Effective Automatic Recognition Systems of Emotion in Speech , 2014 .

[55]  Klaus Krippendorff,et al.  Content Analysis: An Introduction to Its Methodology , 1980 .

[56]  Monica Landoni,et al.  Human Rating of Emotional Expressions - Scales vs. Preferences , 2014, PhyCS.

[57]  Frank Dellaert,et al.  Recognizing emotion in speech , 1996, Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96.

[58]  Mohammad Soleymani,et al.  Affective ranking of movie scenes using physiological signals and content analysis , 2008, MS '08.

[59]  Elmar Nöth,et al.  How to find trouble in communication , 2003, Speech Commun..

[60]  Reza Lotfian,et al.  Practical considerations on the use of preference learning for ranking emotional speech , 2016, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[61]  Ingo Siegert,et al.  Inter-rater reliability for emotion annotation in human–computer interaction: comparison and methodological improvements , 2013, Journal on Multimodal User Interfaces.

[62]  Reza Lotfian,et al.  Retrieving Categorical Emotions Using a Probabilistic Framework to Define Preference Learning Samples , 2016, INTERSPEECH.

[63]  Vanessa Evers,et al.  Automatic temporal ranking of children's engagement levels using multi-modal cues , 2018, Comput. Speech Lang..

[64]  Donald Laming,et al.  The relativity of ‘absolute’ judgements , 1984 .

[65]  Thierry Pun,et al.  DEAP: A Database for Emotion Analysis ;Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.

[66]  Gordon D. A. Brown,et al.  Absolute identification by relative judgment. , 2005, Psychological review.

[67]  Reza Lotfian,et al.  Building a naturalistic emotional speech corpus by retrieving expressive behaviors from existing speech corpora , 2014, INTERSPEECH.

[68]  Sven Ove Hansson What is ceteris paribus preference? , 1996, J. Philos. Log..

[69]  Roddy Cowie,et al.  Automatic recognition of emotion from voice: a rough benchmark , 2000 .

[70]  Zhaocheng Huang,et al.  Detecting the instant of emotion change from speech using a martingale framework , 2016, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[71]  Klaus R. Scherer,et al.  Lost Luggage: A Field Study of Emotion–Antecedent Appraisal , 1997 .

[72]  Arvid Kappas,et al.  Measuring Emotions Online: Expression and Physiology , 2017 .

[73]  Georgios N. Yannakakis,et al.  Preference Learning for Cognitive Modeling: A Case Study on Entertainment Preferences , 2009, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[74]  A. Tversky,et al.  Subjective Probability: A Judgment of Representativeness , 1972 .

[75]  M. Kendall A NEW MEASURE OF RANK CORRELATION , 1938 .

[76]  Carlos Busso,et al.  Defining Emotionally Salient Regions Using Qualitative Agreement Method , 2016, INTERSPEECH.

[77]  Kostas Karpouzis,et al.  The platformer experience dataset , 2015, 2015 International Conference on Affective Computing and Intelligent Interaction (ACII).

[78]  Tao Qin,et al.  LETOR: A benchmark collection for research on learning to rank for information retrieval , 2010, Information Retrieval.

[79]  Louis-Philippe Morency,et al.  Local-global ranking for facial expression intensity estimation , 2017, 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII).

[80]  M. Friedman A Comparison of Alternative Tests of Significance for the Problem of $m$ Rankings , 1940 .

[81]  Roddy Cowie,et al.  Tracing Emotion: An Overview , 2012, Int. J. Synth. Emot..

[82]  Mel W. Khaw,et al.  Normalization is a general neural mechanism for context-dependent decision making , 2013, Proceedings of the National Academy of Sciences.

[83]  Maja Pantic,et al.  Automatic Analysis of Facial Expressions: The State of the Art , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[84]  Carlos Busso,et al.  The Cost of Dichotomizing Continuous Labels for Binary Classification Problems: Deriving a Bayesian-Optimal Classifier , 2017, IEEE Transactions on Affective Computing.

[85]  Robert D. Nowak,et al.  Active Ranking using Pairwise Comparisons , 2011, NIPS.

[86]  Klaus Obermayer,et al.  Support vector learning for ordinal regression , 1999 .

[87]  Andrea Bonarini,et al.  Learning General Preference Models from Physiological Responses in Video Games: How Complex Is It? , 2011, ACII.

[88]  Georgios N. Yannakakis,et al.  Ratings are Overrated! , 2015, Front. ICT.

[89]  Xin Wang,et al.  Modeling transition patterns between events for temporal human action segmentation and classification , 2015, 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[90]  J. Russell Forced-choice response format in the study of facial expression , 1993 .

[91]  Georgios N. Yannakakis Preference learning for affective modeling , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[92]  Georgios N. Yannakakis,et al.  Grounding truth via ordinal annotation , 2015, 2015 International Conference on Affective Computing and Intelligent Interaction (ACII).

[93]  Samuel M. McClure,et al.  Anchors, scales and the relative coding of value in the brain , 2008, Current Opinion in Neurobiology.

[94]  Nebojsa Jojic,et al.  Efficient Ranking from Pairwise Comparisons , 2013, ICML.

[95]  Antonios Liapis,et al.  Towards general models of player affect , 2017, 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII).

[96]  John Deigh Concepts of Emotions in Modern Philosophy and Psychology , 2018 .

[97]  Kostas Karpouzis,et al.  Non-Verbal Feedback on User Interest Based on Gaze Direction and Head Pose , 2007 .

[98]  Antonios Liapis,et al.  Modelling Affect for Horror Soundscapes , 2019, IEEE Transactions on Affective Computing.

[99]  Gordon D. A. Brown,et al.  Decision by sampling , 2006, Cognitive Psychology.

[100]  Mohammad Soleymani,et al.  Queries and tags in affect-based multimedia retrieval , 2009, 2009 IEEE International Conference on Multimedia and Expo.

[101]  Michael Kranzfelder,et al.  New technologies for information retrieval to achieve situational awareness and higher patient safety in the surgical operating room: the MRI institutional approach and review of the literature , 2011, Surgical Endoscopy.

[102]  Emmanuel Dellandréa,et al.  From crowdsourced rankings to affective ratings , 2014, 2014 IEEE International Conference on Multimedia and Expo Workshops (ICMEW).

[103]  Maja Pantic,et al.  Variational Gaussian Process Auto-Encoder for Ordinal Prediction of Facial Action Units , 2016, ACCV.

[104]  Roddy Cowie,et al.  Gtrace: General Trace Program Compatible with EmotionML , 2013, 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.

[105]  Wei Chu,et al.  Preference learning with Gaussian processes , 2005, ICML.

[106]  Tiago H. Falk,et al.  Biometrics and classifier fusion to predict the fun-factor in video gaming , 2016, 2016 IEEE Conference on Computational Intelligence and Games (CIG).

[107]  S. Stevens To Honor Fechner and Repeal His Law , 2008 .

[108]  Reza Lotfian,et al.  Ranking emotional attributes with deep neural networks , 2017, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[109]  Qianhua He,et al.  A survey on emotional semantic image retrieval , 2008, 2008 15th IEEE International Conference on Image Processing.

[110]  S. Shafir,et al.  Context-dependent violations of rational choice in honeybees (Apis mellifera) and gray jays (Perisoreus canadensis) , 2001, Behavioral Ecology and Sociobiology.

[111]  W. Schultz,et al.  Relative reward preference in primate orbitofrontal cortex , 1999, Nature.

[112]  Yi-Hsuan Yang,et al.  Ranking-Based Emotion Recognition for Music Organization and Retrieval , 2011, IEEE Transactions on Audio, Speech, and Language Processing.

[113]  Souhila Kaci,et al.  Working with Preferences: Less Is More , 2011, Cognitive Technologies.

[114]  M. Kohn,et al.  Class and conformity : a study in values , 1970 .

[115]  F. Wilcoxon Individual Comparisons by Ranking Methods , 1945 .

[116]  Vladimir Pavlovic,et al.  Copula Ordinal Regression Framework for Joint Estimation of Facial Action Unit Intensity , 2019, IEEE Transactions on Affective Computing.

[117]  Georgios N. Yannakakis,et al.  Don’t Classify Ratings of Affect; Rank Them! , 2014, IEEE Transactions on Affective Computing.

[118]  J. Cohn,et al.  A Psychometric Evaluation of the Facial Action Coding System for Assessing Spontaneous Expression , 2001 .

[119]  Maja Pantic,et al.  This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. IEEE TRANSACTIONS ON AFFECTIVE COMPUTING , 2022 .

[120]  Amy P Abernethy,et al.  Oncologist communication about emotion during visits with patients with advanced cancer. , 2007, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.

[121]  Roddy Cowie,et al.  FEELTRACE: an instrument for recording perceived emotion in real time , 2000 .

[122]  Antonios Liapis,et al.  A Study on Affect Model Validity: Nominal vs Ordinal Labels , 2018, AffComp@IJCAI.

[123]  James A. Russell,et al.  Adaptation level and the affective appraisal of environments , 1984 .

[124]  Peter Goldie The Mess Inside: Narrative, Emotion, and the Mind , 2012 .

[125]  Georgios N. Yannakakis,et al.  Deep Multimodal Fusion: Combining Discrete Events and Continuous Signals , 2014, ICMI.

[126]  Tie-Yan Liu,et al.  Learning to rank: from pairwise approach to listwise approach , 2007, ICML '07.

[127]  Angeliki Metallinou,et al.  Annotation and processing of continuous emotional attributes: Challenges and opportunities , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[128]  A. Tversky,et al.  The framing of decisions and the psychology of choice. , 1981, Science.

[129]  Antonios Liapis,et al.  Targeting Horror via Level and Soundscape Generation , 2015, AIIDE.

[130]  J. Neumann,et al.  Theory of games and economic behavior , 1945, 100 Years of Math Milestones.

[131]  Georgios N. Yannakakis,et al.  Ranking vs. Preference: A Comparative Study of Self-reporting , 2011, ACII.

[132]  Carlos Busso,et al.  Using neutral speech models for emotional speech analysis , 2007, INTERSPEECH.

[133]  C. Padoa-Schioppa,et al.  Neurons in the orbitofrontal cortex encode economic value , 2006, Nature.

[134]  Navneet Kaur,et al.  Opinion mining and sentiment analysis , 2016, 2016 3rd International Conference on Computing for Sustainable Global Development (INDIACom).

[135]  R. Zajonc Attitudinal effects of mere exposure. , 1968 .

[136]  P. Glimcher,et al.  Reward Value-Based Gain Control: Divisive Normalization in Parietal Cortex , 2011, The Journal of Neuroscience.

[137]  Young Ik Cho,et al.  The Relation Between Culture and Response Styles , 2005 .