AffectButton: A method for reliable and valid affective self-report

In this article we report on a new digital interactive self-report method for the measurement of human affect. The AffectButton (Broekens and Brinkman, 2009. ACII 2009: IEEE) is a button that enables users to provide affective feedback in terms of values on the well-known three affective dimensions of pleasure (valence), arousal and dominance. The AffectButton is an interface component that functions and looks like a medium-sized button. The button presents one dynamically changing iconic facial expression that changes based on the coordinates of the user's pointer in the button. To give affective feedback the user selects the most appropriate expression by clicking the button, effectively enabling 1-click affective self-report on 3 affective dimensions. Here we analyze 5 previously published studies, and 3 novel large-scale studies (n=325, n=202, n=128). Our results show the reliability, validity, and usability of the button for acquiring three types of affective feedback in various domains. The tested domains are holiday preferences, real-time music annotation, emotion words, and textual situation descriptions (ANET). The types of affective feedback tested are preferences, affect attribution to the previously mentioned stimuli, and self-reported mood. All of the subjects tested were Dutch and aged between 15 and 56 years. We end this article with a discussion of the limitations of the AffectButton and of its relevance to areas including recommender systems, preference elicitation, social computing, online surveys, coaching and tutoring, experimental psychology and psychometrics, content annotation, and game consoles.

[1]  R. Reisenzein Pleasure-Arousal Theory and the Intensity of Emotions , 1994 .

[2]  A. Isen,et al.  The Influence of Positive Affect and Visual Access on the Discovery of Integrative Solutions in Bilateral Negotiation , 1986 .

[3]  Clifford Nass,et al.  The media equation - how people treat computers, television, and new media like real people and places , 1996 .

[4]  K. Scherer,et al.  Appraisal processes in emotion: Theory, methods, research. , 2001 .

[5]  K. Scherer,et al.  Appraisal processes in emotion. , 2003 .

[6]  C. Kleinke Gaze and eye contact: a research review. , 1986, Psychological bulletin.

[7]  Nico H. Frijda,et al.  Emotions and Action , 2004 .

[8]  J. Russell,et al.  Evidence for a three-factor theory of emotions , 1977 .

[9]  Willem-Paul Brinkman,et al.  Design and Evaluation of a Virtual Environment for the Treatment of Anger , 2011 .

[10]  Clifford Nass,et al.  Computers that care: investigating the effects of orientation of emotion exhibited by an embodied computer agent , 2005, Int. J. Hum. Comput. Stud..

[11]  J. Russell Core affect and the psychological construction of emotion. , 2003, Psychological review.

[12]  Koen V. Hindriks,et al.  Proceedings of the 1st International Working Conference on Human Factors and Computational Models in Negotiation , 2008 .

[13]  A. Damasio Descartes' error: emotion, reason, and the human brain. avon books , 1994 .

[14]  James A. Russell,et al.  A Verbal Measure of Information Rate for Studies in Environmental Psychology , 1974 .

[15]  L. F. Barrett,et al.  Handbook of Emotions , 1993 .

[16]  Catholijn M. Jonker,et al.  Designing interfaces for explicit preference elicitation: a user-centered investigation of preference representation and elicitation process , 2011, User Modeling and User-Adapted Interaction.

[17]  Chun-Chia Lee,et al.  AIMED- A Personalized TV Recommendation System , 2006, EuroITV.

[18]  Joost Broekens,et al.  In Defense of Dominance: PAD Usage in Computational Representations of Affect , 2012, Int. J. Synth. Emot..

[19]  Timothy W. Bickmore,et al.  Establishing and maintaining long-term human-computer relationships , 2005, TCHI.

[20]  L. Hedges,et al.  Introduction to Meta‐Analysis , 2009, International Coaching Psychology Review.

[21]  Michael Kipp,et al.  Designing Emotions , 2011, KI - Künstliche Intelligenz.

[22]  Joost Broekens,et al.  Emotion and Reinforcement: Affective Facial Expressions Facilitate Robot Learning , 2007, Artifical Intelligence for Human Computing.

[23]  M. Bradley,et al.  Affective reactions to acoustic stimuli. , 2000, Psychophysiology.

[24]  Jaak Panksepp,et al.  Toward a general psychobiological theory of emotions , 1982, Behavioral and Brain Sciences.

[25]  Ben J. A. Kröse,et al.  The Influence of a Robot's Social Abilities on Acceptance by Elderly Users , 2006, ROMAN 2006 - The 15th IEEE International Symposium on Robot and Human Interactive Communication.

[26]  Steve Benford,et al.  EmoPlayer: A media player for video clips with affective annotations , 2008, Interact. Comput..

[27]  Ana Paiva,et al.  Affective Interactions: Toward a New Generation of Computer Interfaces? , 2000, IWAI.

[28]  Kristina Höök,et al.  eMoto: emotionally engaging interaction , 2004, Personal and Ubiquitous Computing.

[29]  Jonathan L. Herlocker,et al.  Evaluating collaborative filtering recommender systems , 2004, TOIS.

[30]  Petra Sundström,et al.  Exploring the Affective Loop , 2005 .

[31]  G. Dreisbach,et al.  How positive affect modulates cognitive control: reduced perseveration at the cost of increased distractibility. , 2004, Journal of experimental psychology. Learning, memory, and cognition.

[32]  Wietske Visser,et al.  Multi-angle view on preference elicitation for negotiation support systems , 2008, HuCom.

[33]  Hugh Coolican Research Methods and Statistics in Psychology , 1993 .

[34]  E. Hudlicka AFFECTIVE COMPUTING FOR GAME DESIGN , 2008 .

[35]  E. Cashdan Smiles, Speech, and Body Posture: How Women and Men Display Sociometric Status and Power , 1998 .

[36]  L. Thompson,et al.  Affect from the top down: How powerful individuals' positive affect shapes negotiations , 2004 .

[37]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2009, IEEE Trans. Pattern Anal. Mach. Intell..

[38]  James C. Lester,et al.  Modeling and evaluating empathy in embodied companion agents , 2007, Int. J. Hum. Comput. Stud..

[39]  Kate S. Hone,et al.  Empathic agents to reduce user frustration: The effects of varying agent characteristics , 2006, Interact. Comput..

[40]  Alexandru Iosup,et al.  TRIBLER: a social-based peer-to-peer system: Research Articles , 2008 .

[41]  Joseph B. Walther,et al.  The Impacts of Emoticons on Message Interpretation in Computer-Mediated Communication , 2001 .

[42]  Jonathan Klein,et al.  Computers that recognise and respond to user emotion: theoretical and practical implications , 2002, Interact. Comput..

[43]  Eva Hudlicka,et al.  To feel or not to feel: The role of affect in human-computer interaction , 2003, Int. J. Hum. Comput. Stud..

[44]  Joost Broekens,et al.  AffectButton: Towards a standard for dynamic affective user feedback , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[45]  James J. Kellaris,et al.  An exploratory investigation of responses elicited by music varying in tempo, tonality, and texture , 1993 .

[46]  Rafael A. Calvo,et al.  Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications , 2010, IEEE Transactions on Affective Computing.

[47]  Peter H. M. P. Roelofsma,et al.  The Effect of Dominance Manipulation on the Perception and Believability of an Emotional Expression , 2014, Emotion Modeling.

[48]  Hillary Anger Elfenbein,et al.  On the universality and cultural specificity of emotion recognition: a meta-analysis. , 2002, Psychological bulletin.

[49]  Cristina Conati,et al.  Affective interactions: the computer in the affective loop , 2005, IUI.

[50]  Kristina Höök,et al.  Using the Sensual Evaluation Instrument , 2009, Digit. Creativity.

[51]  G. A. Mendelsohn,et al.  Affect grid : A single-item scale of pleasure and arousal , 1989 .

[52]  Catholijn M. Jonker,et al.  Factors Influencing User Motivation for Giving Online Preference Feedback , 2010 .

[53]  M. Bradley,et al.  Measuring emotion: the Self-Assessment Manikin and the Semantic Differential. , 1994, Journal of behavior therapy and experimental psychiatry.

[54]  Michael W. Eysenck,et al.  Psychology: An International Perspective , 2004 .

[55]  A. Isen,et al.  Positive affect facilitates creative problem solving. , 1987, Journal of personality and social psychology.

[56]  Brian A. Maurer,et al.  Body size, ecological dominance and Cope's rule , 1986, Nature.

[57]  Maja Pantic,et al.  Facial action recognition for facial expression analysis from static face images , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[58]  A. Mehrabian Basic Dimensions For A General Psychological Theory , 1980 .

[59]  C. Bartneck,et al.  In your face, robot! The influence of a character's embodiment on how users perceive its emotional expressions , 2004 .

[60]  Egon L. van den Broek,et al.  Tune in to your emotions: a robust personalized affective music player , 2012, User Modeling and User-Adapted Interaction.

[61]  G. Weisfeld,et al.  Erectness of posture as an indicator of dominance or success in humans , 1982 .

[62]  Alan J. Dix,et al.  Affective Videogames and Modes of Affective Gaming: Assist Me, Challenge Me, Emote Me (ACE) , 2005, DiGRA Conference.

[63]  Ana Paiva,et al.  Caring for agents and agents that care: building empathic relations with synthetic agents , 2004, Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems, 2004. AAMAS 2004..

[64]  Joost Broekens,et al.  Modeling the Experience of Emotion , 2009, Int. J. Synth. Emot..

[65]  K. Scherer,et al.  Evidence for universality and cultural variation of differential emotion response patterning. , 1994, Journal of personality and social psychology.

[66]  Kristina Höök,et al.  Affective Loop Experiences - What Are They? , 2008, PERSUASIVE.

[67]  Georgios N. Yannakakis,et al.  Real-Time Game Adaptation for Optimizing Player Satisfaction , 2009, IEEE Transactions on Computational Intelligence and AI in Games.

[68]  Joseph E LeDoux The emotional brain , 1996 .

[69]  Cynthia Breazeal,et al.  Emotion and sociable humanoid robots , 2003, Int. J. Hum. Comput. Stud..

[70]  Jacob Cohen,et al.  A power primer. , 1992, Psychological bulletin.

[71]  Andrew Clement,et al.  Proceedings of the eighth conference on Participatory design: Artful integration: interweaving media, materials and practices - Volume 1 , 2004 .

[72]  Agneta H. Fischer,et al.  Feelings and Emotions: The Amsterdam Symposium , 2004 .

[73]  Muriel Garreta-Domingo,et al.  Ten emotion heuristics: guidelines for assessing the user's affective dimension easily and cost-effectively , 2007 .

[74]  Philippe A. Palanque,et al.  Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , 2014, International Conference on Human Factors in Computing Systems.

[75]  Sriram Subramanian,et al.  Talking about tactile experiences , 2013, CHI.

[76]  Minna Isomursu,et al.  Experimental evaluation of five methods for collecting emotions in field settings with mobile applications , 2007, Int. J. Hum. Comput. Stud..

[77]  Minna Isomursu,et al.  Experience clip: method for user participation and evaluation of mobile concepts , 2004, PDC 04.

[78]  Joost Broekens,et al.  Real time labeling of affect in music using the affectbutton , 2010, AFFINE '10.

[79]  N. Frijda The laws of emotion. , 1988, The American psychologist.

[80]  Janet C. Read,et al.  Validating the Fun Toolkit: an instrument for measuring children’s opinions of technology , 2008, Cognition, Technology & Work.

[81]  Mohammad Soleymani,et al.  Crowdsourcing for Affective Annotation of Video: Development of a Viewer-reported Boredom Corpus , 2010 .

[82]  J. Alfredo Sánchez,et al.  Conveying mood and emotion in instant messaging by using a two-dimensional model for affective states , 2006, IHC '06.

[83]  Rosalind W. Picard Affective computing: challenges , 2003, Int. J. Hum. Comput. Stud..

[84]  G. Band,et al.  In the Mood for Adaptation , 2010, Psychological science.

[85]  Stacy Marsella,et al.  Tears and fears: modeling emotions and emotional behaviors in synthetic agents , 2001, AGENTS '01.

[86]  K. Loewenthal An Introduction to Psychological Tests and Scales , 2018 .

[87]  P. Tunstall,et al.  Teacher Feedback to Young Children in Formative Assessment: a typology , 1996 .

[88]  Pieter Desmet,et al.  Measuring Emotion: Development and Application of an Instrument to Measure Emotional Responses to Products , 2005, Funology.

[89]  Arthur C. Graesser,et al.  Automatic detection of learner’s affect from conversational cues , 2008, User Modeling and User-Adapted Interaction.

[90]  G. Fricchione Descartes’ Error: Emotion, Reason and the Human Brain , 1995 .

[91]  A. Chaudhuri,et al.  The Many Faces of a Neutral Face: Head Tilt and Perception of Dominance and Emotion , 2003 .

[92]  J. Russell A circumplex model of affect. , 1980 .

[93]  Mohan S. Kankanhalli,et al.  Proceedings of the 13th annual ACM international conference on Multimedia , 2005, MM 2005.

[94]  Péter Pongrácz,et al.  Human listeners are able to classify dog (Canis familiaris) barks recorded in different situations. , 2005, Journal of comparative psychology.

[95]  Connor Dickie,et al.  Emotionally reactive television , 2007, IUI '07.

[96]  FrijdaMansteadBem Emotions and Beliefs: How Feelings Influence Thoughts , 2000 .

[97]  Gerben A. van Kleef,et al.  Power and emotion in negotiation: Power moderates the interpersonal effects of anger and happiness on concession making , 2006 .

[98]  P. Ekman,et al.  Unmasking the Face: A Guide to Recognizing Emotions From Facial Expressions , 1975 .

[99]  E. Rolls Précis of The brain and emotion. , 2000, The Behavioral and brain sciences.

[100]  Mor Naaman,et al.  Why we tag: motivations for annotation in mobile and online media , 2007, CHI.

[101]  Andrew Ortony,et al.  The Cognitive Structure of Emotions , 1988 .

[102]  Jun Wang,et al.  TRIBLER: a social‐based peer‐to‐peer system , 2008, IPTPS.

[103]  Nicu Sebe,et al.  Affective multimodal human-computer interaction , 2005, ACM Multimedia.

[104]  C. Osgood Dimensionality of the semantic space for communication via facial expressions. , 1966, Scandinavian journal of psychology.

[105]  H. Chad Lane,et al.  Teaching Negotiation Skills through Practice and Reflection with Virtual Humans , 2006, Simul..

[106]  Daantje Derks,et al.  The role of emotion in computer-mediated communication: A review , 2008, Comput. Hum. Behav..

[107]  Robin D. Burke,et al.  Hybrid Recommender Systems: Survey and Experiments , 2002, User Modeling and User-Adapted Interaction.

[108]  D. Watson,et al.  Development and validation of brief measures of positive and negative affect: the PANAS scales. , 1988, Journal of personality and social psychology.

[109]  Karen Gasper,et al.  Attending to the Big Picture: Mood and Global Versus Local Processing of Visual Information , 2002, Psychological science.

[110]  Rosalind W. Picard Affective Computing , 1997 .

[111]  Arthur C. Graesser,et al.  Toward an Affect-Sensitive AutoTutor , 2007, IEEE Intelligent Systems.

[112]  Simon Brown,et al.  Affective gaming: measuring emotion through the gamepad , 2003, CHI Extended Abstracts.

[113]  Stacy Marsella,et al.  Modeling the cognitive antecedents and consequences of emotion , 2009, Cognitive Systems Research.

[114]  Gilbert Cockton,et al.  CHI '03 Extended Abstracts on Human Factors in Computing Systems , 2003, CHI 2003.

[115]  S. Paradiso,et al.  Book Review: Affective Neuroscience: The Foundations of Human and Animal Emotions , 2000 .

[116]  Kristina Höök,et al.  The sensual evaluation instrument: developing an affective evaluation tool , 2006, CHI.

[117]  Willem-Paul Brinkman,et al.  The therapist user interface of a virtual reality exposure therapy system in the treatment of fear of flying , 2010, Interact. Comput..

[118]  Roddy Cowie,et al.  FEELTRACE: an instrument for recording perceived emotion in real time , 2000 .

[119]  J. Russell Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies. , 1994, Psychological bulletin.

[120]  K. Vohs,et al.  How Emotion Shapes Behavior: Feedback, Anticipation, and Reflection, Rather Than Direct Causation , 2007, Personality and social psychology review : an official journal of the Society for Personality and Social Psychology, Inc.

[121]  Janet C. Read,et al.  Child computer interaction: advances in methodological research , 2007, Cognition, Technology & Work.

[122]  Kristina Höök,et al.  The sensual evaluation instrument: Developing a trans-cultural self-report measure of affect , 2007, Int. J. Hum. Comput. Stud..

[123]  Kasia Muldner,et al.  Affective Tutors: Automatic Detection of and Response to Student Emotion , 2010, Advances in Intelligent Tutoring Systems.

[124]  Gareth R. Jones,et al.  The Role of Affect in Cross-Cultural Negotiations , 1998 .

[125]  P. Lang International affective picture system (IAPS) : affective ratings of pictures and instruction manual , 2005 .

[126]  Jennifer C. Britton,et al.  Facial expressions and complex IAPS pictures: Common and differential networks , 2006, NeuroImage.

[127]  A. Manstead,et al.  Social functions of emotion , 2008 .

[128]  Wijnand A. IJsselsteijn,et al.  Intimate Heartbeats: Opportunities for Affective Communication Technology , 2010, IEEE Transactions on Affective Computing.

[129]  C. McGarty,et al.  Research Methods and Statistics in Psychology , 2003 .

[130]  Arthur C. Graesser,et al.  AutoTutor: an intelligent tutoring system with mixed-initiative dialogue , 2005, IEEE Transactions on Education.

[131]  Hilary Johnson,et al.  Developing affective educational software products: Sorémo, a new method for capturing emotional states , 2009 .

[132]  P. Ekman Facial expression and emotion. , 1993, The American psychologist.