The Affective Computing Approach to Affect Measurement

Affective computing (AC) adopts a computational approach to study affect. We highlight the AC approach towards automated affect measures that jointly model machine-readable physiological/behavioral signals with affect estimates as reported by humans or experimentally elicited. We describe the conceptual and computational foundations of the approach followed by two case studies: one on discrimination between genuine and faked expressions of pain in the lab, and the second on measuring nonbasic affect in the wild. We discuss applications of the measures, analyze measurement accuracy and generalizability, and highlight advances afforded by computational tipping points, such as big data, wearable sensing, crowdsourcing, and deep learning. We conclude by advocating for increasing synergies between AC and affective science and offer suggestions toward that direction.

[1]  Ryan Shaun Joazeiro de Baker,et al.  Using Video to Automatically Detect Learner Affect in Computer-Enabled Classrooms , 2016, TIIS.

[2]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[3]  Laszlo A. Jeni,et al.  Spontaneous facial expression in unscripted social interactions can be measured automatically , 2015, Behavior research methods.

[4]  Ryan Shaun Joazeiro de Baker,et al.  Accuracy vs. Availability Heuristic in Multimodal Affect Detection in the Wild , 2015, ICMI.

[5]  Jacqueline Kory Westlund,et al.  Motion Tracker: Camera-Based Monitoring of Bodily Movements Using Motion Silhouettes , 2015, PloS one.

[6]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[7]  Sidney K. D'Mello,et al.  A Review and Meta-Analysis of Multimodal Affect Detection Systems , 2015, ACM Comput. Surv..

[8]  Daniel McDuff,et al.  Predicting Ad Liking and Purchase Intent: Large-Scale Analysis of Facial Responses to Ads , 2014, IEEE Transactions on Affective Computing.

[9]  Rafael A. Calvo,et al.  Research and Development Tools in Affective Computing , 2015 .

[10]  Daniel McDuff,et al.  Crowdsourcing Techniques for Affective Computing , 2015 .

[11]  Sidney K. D'Mello,et al.  Affect Elicitation for Affective Computing , 2015 .

[12]  Arthur C. Graesser,et al.  Feeling, Thinking, and Computing with Affect-Aware Learning Technologies , 2015 .

[13]  Roddy Cowie,et al.  Ethical Issues in Affective Computing , 2015 .

[14]  J. Gratch,et al.  The Oxford Handbook of Affective Computing , 2014 .

[15]  B. Mesquita,et al.  Emotions in Context: A Sociodynamic Model of Emotions , 2014 .

[16]  L. F. Barrett.  The Conceptual Act Theory: A Précis , 2014 .

[17]  Daniel Neiberg,et al.  Evidence for cultural dialects in vocal emotion expression: acoustic classification within and across five nations. , 2014, Emotion.

[18]  M. Bartlett,et al.  Automatic Decoding of Facial Movements Reveals Deceptive Pain Expressions , 2014, Current Biology.

[19]  S. D’Mello.  A selective meta-analysis on the relative incidence of discrete affective states during learning with technology , 2013 .

[20]  Fernando De la Torre,et al.  Facing Imbalanced Data--Recommendations for the Use of Performance Metrics , 2013, 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.

[21]  Philip A Kragel,et al.  Multivariate pattern classification reveals autonomic and experiential representations of discrete emotions. , 2013, Emotion.

[22]  Arvid Kappas,et al.  Social regulation of emotion: messy layers , 2013, Front. Psychology.

[23]  Joyce H. D. M. Westerink,et al.  Directing Physiology and Mood through Music: Validation of an Affective Music Player , 2013, IEEE Transactions on Affective Computing.

[24]  Pedro M. Domingos.  A few useful things to know about machine learning , 2012, Commun. ACM.

[25]  Klaus R. Scherer,et al.  A psycho-ethological approach to social signal processing , 2012, Cognitive Processing.

[26]  Ki H. Chon,et al.  Physiological Parameter Monitoring from Optical Recordings With a Mobile Phone , 2012, IEEE Transactions on Biomedical Engineering.

[27]  K. Scherer,et al.  In the eye of the beholder? Universality and cultural specificity in the expression and perception of emotion. , 2011, International journal of psychology : Journal international de psychologie.

[28]  Ira J. Roseman.  Emotional Behaviors, Emotivational Goals, Emotion Strategies: Multiple Levels of Organization Integrate Variable and Consistent Responses , 2011 .

[29]  Gwen Littlewort,et al.  The computer expression recognition toolbox (CERT) , 2011, Face and Gesture 2011.

[30]  Etienne B. Roesch,et al.  A Blueprint for Affective Computing: A Sourcebook and Manual , 2010 .

[31]  C. Izard.  The Many Meanings/Aspects of Emotion: Definitions, Functions, Activation, and Regulation , 2010 .

[32]  Rosalind W. Picard.  Emotion Research by the People, for the People , 2010 .

[33]  Rosalind W. Picard,et al.  Non-contact, automated cardiac pulse measurements using video imaging and blind source separation , 2022 .

[34]  Oleg V. Komogortsev,et al.  Real-time eye gaze tracking with an unmodified commodity webcam employing a neural network , 2010, CHI Extended Abstracts.

[35]  Jadwiga Indulska,et al.  A survey of context modelling and reasoning techniques , 2010, Pervasive Mob. Comput..

[36]  Arvid Kappas,et al.  Smile When You Read This, Whether You Like It or Not: Conceptual Challenges to Affect Detection , 2010, IEEE Transactions on Affective Computing.

[37]  Rafael A. Calvo,et al.  Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications , 2010, IEEE Transactions on Affective Computing.

[38]  P. Petta,et al.  Computational models of emotion , 2010 .

[39]  Hatice Gunes,et al.  Automatic, Dimensional and Continuous Emotion Recognition , 2010, Int. J. Synth. Emot..

[40]  Maja Pantic,et al.  Social signal processing: Survey of an emerging domain , 2009, Image Vis. Comput..

[41]  Nir Friedman,et al.  Probabilistic Graphical Models - Principles and Techniques , 2009 .

[42]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  John J. B. Allen,et al.  The handbook of emotion elicitation and assessment , 2007 .

[44]  Mark Goadrich,et al.  The relationship between Precision-Recall and ROC curves , 2006, ICML.

[45]  R. Dodhia.  A Review of Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences (3rd ed.) , 2005 .

[46]  Marc D. Lewis.  Bridging emotion theory and neurobiology through dynamic systems modeling , 2005, Behavioral and Brain Sciences.

[47]  Nir Friedman,et al.  Bayesian Network Classifiers , 1997, Machine Learning.

[48]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[49]  Hillary Anger Elfenbein,et al.  On the universality and cultural specificity of emotion recognition: a meta-analysis. , 2002, Psychological bulletin.

[50]  Jennifer Healey,et al.  Toward Machine Emotional Intelligence: Analysis of Affective Physiological State , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[51]  E. Vesterinen,et al.  Affective Computing , 2009, Encyclopedia of Biometrics.

[52]  Rosalind W. Picard.  Affective computing: (526112012-054) , 1997 .

[53]  K. Craig,et al.  Subjective judgments of deception in pain expression: accuracy and errors , 1996, Pain.

[54]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[55]  F Dejong Estienne,et al.  [Voice and emotion]. , 1991, Revue de laryngologie - otologie - rhinologie.

[56]  J. Hanley,et al.  The meaning and use of the area under a receiver operating characteristic (ROC) curve. , 1982, Radiology.

[57]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[58]  Jacob Cohen,et al.  Applied multiple regression/correlation analysis for the behavioral sciences , 1979 .

[59]  R. Graczyk.  The eye. , 1955, Radiography.