Grounding truth via ordinal annotation

The question of how to best annotate affect within available content has been a milestone challenge for affective computing. Appropriate methods and tools addressing that question can provide better estimations of the ground truth which, in turn, may lead to more efficient affect detection and more reliable models of affect. This paper introduces a rank-based real-time annotation tool, we name AffectRank, and compares it against the popular rating-based real-time FeelTrace tool through a proof-of-concept video annotation experiment. Results obtained suggest that the rank-based (ordinal) annotation approach proposed yields significantly higher inter-rater reliability and, thereby, approximation of the underlying ground truth. The key findings of the paper demonstrate that the current dominant practice in continuous affect annotation via rating-based labeling is detrimental to advancements in the field of affective computing.

[1]  Yi-Hsuan Yang,et al.  Ranking-Based Emotion Recognition for Music Organization and Retrieval , 2011, IEEE Transactions on Audio, Speech, and Language Processing.

[2]  Georgios N. Yannakakis,et al.  Ranking vs. Preference: A Comparative Study of Self-reporting , 2011, ACII.

[3]  Roddy Cowie,et al.  Real life emotions in French and English TV video clips: an integrated annotation protocol combining continuous and discrete approaches , 2006, LREC.

[4]  M. Kendall.  A NEW MEASURE OF RANK CORRELATION , 1938 .

[5]  Roddy Cowie,et al.  Gtrace: General Trace Program Compatible with EmotionML , 2013, 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.

[6]  Georgios N. Yannakakis.  Preference learning for affective modeling , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[7]  J. Russell.  A circumplex model of affect. , 1980 .

[8]  W. Kruskal,et al.  Use of Ranks in One-Criterion Variance Analysis , 1952 .

[9]  Maja Pantic,et al.  This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. IEEE TRANSACTIONS ON AFFECTIVE COMPUTING , 2022 .

[10]  Georgios N. Yannakakis,et al.  The Preference Learning Toolbox , 2015, ArXiv.

[11]  Angeliki Metallinou,et al.  Annotation and processing of continuous emotional attributes: Challenges and opportunities , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[12]  J. Cohn,et al.  Infant Smiling Dynamics and Perceived Positive Emotion , 2008, Journal of nonverbal behavior.

[13]  Roddy Cowie,et al.  Describing the emotional states that are expressed in speech , 2003, Speech Commun..

[14]  Klaus Krippendorff,et al.  Content Analysis: An Introduction to Its Methodology , 1980 .

[15]  Andrea Bonarini,et al.  Modeling enjoyment preference from physiological responses in a car racing game , 2010, Proceedings of the 2010 IEEE Conference on Computational Intelligence and Games.

[16]  Roddy Cowie,et al.  FEELTRACE: an instrument for recording perceived emotion in real time , 2000 .

[17]  Athanasia Zlatintsi,et al.  A supervised approach to movie emotion tracking , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[18]  Eyke Hüllermeier,et al.  Preference Learning , 2005, Künstliche Intell..

[19]  Eyke Hllermeier,et al.  Preference Learning , 2010 .

[20]  Georgios N. Yannakakis,et al.  Ratings are Overrated! , 2015, Front. ICT.

[21]  F. Wilcoxon.  Individual Comparisons by Ranking Methods , 1945 .

[22]  M. Friedman.  A Comparison of Alternative Tests of Significance for the Problem of $m$ Rankings , 1940 .

[23]  L. Cronbach.  Coefficient alpha and the internal structure of tests , 1951 .

[24]  Eckart Altenmüller,et al.  EMuJoy: Software for continuous measurement of perceived emotions in music , 2007, Behavior research methods.

[25]  Yoshua Bengio,et al.  Learning deep physiological models of affect , 2013, IEEE Computational Intelligence Magazine.

[26]  Georgios N. Yannakakis,et al.  Don’t Classify Ratings of Affect; Rank Them! , 2014, IEEE Transactions on Affective Computing.