Affectemes and allaffects: A novel approach to coding user emotional expression during interactive experiences

The potential importance of human affect during human - computer interaction (HCI) is becoming increasingly well recognised. However, measuring and analysing affective behaviour is problematic. Physiological indicators reveal only some, sometimes ambiguous information. Video analysis and existing coding schemes are notoriously lengthy and complex, and examine only certain aspects of affect. This paper describes the development of a practical methodology to assess user affect, as displayed by emotional expressions. Interaction analysis techniques were used to identify discrete affective messages 'affectemes' and their components. This paper explains the rationale for this approach and demonstrates how it can be applied in practice. Preliminary evidence for its efficacy and reliability is also presented.

[1]  A J Shinkfield,et al.  Recognition of facial emotional expressions from moving and static displays by individuals with mental retardation. , 1999, American journal of mental retardation : AJMR.

[2]  Randolph R. Cornelius,et al.  The science of emotion: Research and tradition in the psychology of emotion. , 1997 .

[3]  Thomas Wehrle,et al.  Emotion research and AI: Some theoretical and technical issues* , 2022 .

[4]  Robert Biswas-Diener,et al.  Emotions Across Cultures and Methods , 2004 .

[5]  P. Ekman Emotions Revealed: Recognizing Faces and Feelings to Improve Communication and Emotional Life , 2003 .

[6]  Hervé Abdi,et al.  Processing Faces and Facial Expressions , 2003, Neuropsychology Review.

[7]  D. Norman Emotional design : why we love (or hate) everyday things , 2004 .

[8]  S. Kaiser,et al.  Automated coding of facial behavior in human-computer interactions with facs , 1992 .

[9]  Robert D. Ward,et al.  Affective computing: problems, reactions and intentions , 2004, Interact. Comput..

[10]  G. Bonanno,et al.  Emotions : current issues and future directions , 2001 .

[11]  Irwin Silverman,et al.  Sex Related Factors in the Perception of Threatening Facial Expressions , 2002 .

[12]  S. Bunce,et al.  Repressive coping style and the experience and recall of emotion: a naturalistic study of daily affect. , 1996, Journal of personality.

[13]  J. Singer,et al.  Cognitive, social, and physiological determinants of emotional state. , 1962 .

[14]  J. Tropman,et al.  The Managed Heart: Commercialization of Human Feeling , 1984 .

[15]  Patricia Wallace The Psychology of the Internet , 1999 .

[16]  A. Hochschild,et al.  The Managed Heart: Commercialization of Human Feeling. , 1985 .

[17]  Gavriel Salvendy,et al.  A personal perspective on behaviour and information technology: A 20-year progress and future trend , 2001, Behav. Inf. Technol..

[18]  A. Cruttenden Gimson's Pronunciation of English , 1994 .

[19]  Michael Kipp,et al.  Gesture generation by imitation: from human behavior to computer character animation , 2005 .

[20]  Peter Robinson,et al.  Gestures and multimodal input , 1999, Behav. Inf. Technol..

[21]  Robert D. Ward,et al.  Physiological responses to different WEB page designs , 2003, Int. J. Hum. Comput. Stud..

[22]  P. Ekman,et al.  What the face reveals : basic and applied studies of spontaneous expression using the facial action coding system (FACS) , 2005 .

[23]  Roger Bakeman,et al.  Analyzing Interaction: Sequential Analysis with SDIS and GSEQ , 1995 .

[24]  In A. H. Fischer Emotional Episodes, Facial Expressions, and Reported Feelings in Human-computer Interactions , 1998 .

[25]  D. Kirsch The Sentic Mouse: Developing a tool for Measuring Emotional Valence , 1997 .

[26]  C. Darwin The Expression of the Emotions in Man and Animals , .

[27]  Ann Hutchinson Guest Labanotation: The System of Analyzing and Recording Movement , 1987 .

[28]  Amy L. Parsons,et al.  Emotional Design: Why We Love (or Hate) Everyday Things , 2006 .

[29]  Madeleine Bunting Willing Slaves: How the Overwork Culture is Ruling Our Lives , 2004 .

[30]  Gavriel Salvendy,et al.  Intranets and Organizational Learning: A Research and Development Agenda , 2002, Int. J. Hum. Comput. Interact..

[31]  Mark Blythe,et al.  Funology:From Usability to Enjoyment , 2003 .

[32]  E. Vesterinen,et al.  Affective Computing , 2009, Encyclopedia of Biometrics.

[33]  Clifford Nass,et al.  The media equation - how people treat computers, television, and new media like real people and places , 1996 .

[34]  Peter C. Wright,et al.  Funology: from usability to enjoyment , 2005 .

[35]  D. Dutton,et al.  Some evidence for heightened sexual attraction under conditions of high anxiety. , 1974, Journal of personality and social psychology.

[36]  M. Bradley,et al.  Measuring emotion: the Self-Assessment Manikin and the Semantic Differential. , 1994, Journal of behavior therapy and experimental psychiatry.

[37]  Harrison Si,et al.  Handbook of Research Methods in Social and Personality Psychology: Author Index , 2013 .

[38]  M. Lombard,et al.  Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability , 2002 .

[39]  M. Buonocore,et al.  Posterior cingulate cortex activation by emotional words: fMRI evidence from a valence decision task , 2003, Human brain mapping.

[40]  Jonathan Klein,et al.  Frustrating the user on purpose: a step toward building an affective computer , 2002, Interact. Comput..

[41]  Cheryl Campanella Bracken,et al.  Practical Resources for Assessing and Reporting Intercoder Reliability in Content Analysis Research Projects , 2005 .

[42]  Hillary Anger Elfenbein,et al.  Emotional intelligence and the recognition of emotion from facial expressions. , 2002 .

[43]  Rosalind W. Picard Affective computing: (526112012-054) , 1997 .

[44]  Kate S. Hone,et al.  E-motional advantage: performance and satisfaction gains with affective computing , 2005, CHI Extended Abstracts.

[45]  Roger Bakeman,et al.  Observing Interaction: An Introduction to Sequential Analysis , 1986 .