Automatic detection of learner’s affect from conversational cues

We explored the reliability of detecting a learner’s affect from conversational features extracted from interactions with AutoTutor, an intelligent tutoring system (ITS) that helps students learn by holding a conversation in natural language. Training data were collected in a learning session with AutoTutor, after which the affective states of the learner were rated by the learner, a peer, and two trained judges. Inter-rater reliability scores indicated that the classifications of the trained judges were more reliable than the novice judges. Seven data sets that temporally integrated the affective judgments with the dialogue features of each learner were constructed. The first four datasets corresponded to the judgments of the learner, a peer, and two trained judges, while the remaining three data sets combined judgments of two or more raters. Multiple regression analyses confirmed the hypothesis that dialogue features could significantly predict the affective states of boredom, confusion, flow, and frustration. Machine learning experiments indicated that standard classifiers were moderately successful in discriminating the affective states of boredom, confusion, flow, frustration, and neutral, yielding a peak accuracy of 42% with neutral (chance = 20%) and 54% without neutral (chance = 25%). Individual detections of boredom, confusion, flow, and frustration, when contrasted with neutral affect, had maximum accuracies of 69, 68, 71, and 78%, respectively (chance = 50%). The classifiers that operated on the emotion judgments of the trained judges and combined models outperformed those based on judgments of the novices (i.e., the self and peer). Follow-up classification analyses that assessed the degree to which machine-generated affect labels correlated with affect judgments provided by humans revealed that human-machine agreement was on par with novice judges (self and peer) but quantitatively lower than trained judges. We discuss the prospects of extending AutoTutor into an affect-sensing ITS.

[1]  Carolyn Penstein Rosé,et al.  Spoken Versus Typed Human and Computer Dialogue Tutoring , 2006, Int. J. Artif. Intell. Educ..

[2]  J. Connell,et al.  What motivates children's behavior and emotion? Joint effects of perceived control and autonomy in the academic domain. , 1993, Journal of personality and social psychology.

[3]  W. DeKeseredy,et al.  Future directions , 2005, Psychiatric Quarterly.

[4]  Louis ten Bosch,et al.  Emotions, speech and the ASR framework , 2003, Speech Commun..

[5]  Scotty D. Craig,et al.  Affect and learning: An exploratory look into the role of affect in learning with AutoTutor , 2004 .

[6]  L. Pessoa,et al.  Positive emotions broaden the scope of attention and thought‐action repertoires , 2005, Cognition & emotion.

[7]  L. Rothkrantz,et al.  Toward an affect-sensitive multimodal human-computer interaction , 2003, Proc. IEEE.

[8]  Alex Pentland,et al.  LAFTER: a real-time face and lips tracker with facial expression recognition , 2000, Pattern Recognit..

[9]  Lars Vavik,et al.  Facilitating Discovery Learning in Computer-Based Simulation Environments , 1995 .

[10]  A. Graesser,et al.  Improving an intelligent tutor ’ s comprehension of students with Latent Semantic Analysis ∗ , 1999 .

[11]  John R. Anderson,et al.  Cognitive Tutors: Lessons Learned , 1995 .

[12]  Brent A. Olde,et al.  How does one know whether a person understands a device? The quality of the questions the person asks when the device breaks down. , 2003 .

[13]  Paul J. Silvia,et al.  Can positive affect induce self-focused attention? Methodological and measurement issues , 2002 .

[14]  Scotty D. Craig,et al.  Integrating Affect Sensors in an Intelligent Tutoring System , 2004 .

[15]  Philip H. Mirvis.  Flow: The Psychology of Optimal Experience , 1991 .

[16]  Mitsuru Ishizuka,et al.  THE EMPATHIC COMPANION: A CHARACTER-BASED INTERFACE THAT ADDRESSES USERS' AFFECTIVE STATES , 2005, Appl. Artif. Intell..

[17]  Georgios N. Yannakakis,et al.  Entertainment capture through heart rate activity in physical interactive playgrounds , 2008, User Modeling and User-Adapted Interaction.

[18]  Mark R. Lepper,et al.  The wisdom of practice: Lessons learned from the study of highly effective tutors. , 2002 .

[19]  Cecilia Ovesdotter Alm,et al.  Perceptions of emotions in expressive storytelling , 2005, INTERSPEECH.

[20]  Elmar Nöth,et al.  Private emotions versus social interaction: a data-driven approach towards analysing emotion in speech , 2008, User Modeling and User-Adapted Interaction.

[21]  Colin Robson,et al.  Real World Research: A Resource for Social Scientists and Practitioner-Researchers , 1993 .

[22]  Jack Mostow,et al.  Adding Human-Provided Emotional Scaffolding to an Automated Reading Tutor That Listens Increases Student Persistence , 2002, Intelligent Tutoring Systems.

[23]  Kurt VanLehn,et al.  Andes: A Coached Problem Solving Environment for Physics , 2000, Intelligent Tutoring Systems.

[24]  D. Goleman.  Emotional Intelligence. New York (Bantam) 1995. , 1995 .

[25]  Arthur C. Graesser,et al.  Using Latent Semantic Analysis to Evaluate the Contributions of Students in AutoTutor , 2000, Interact. Learn. Environ..

[26]  Arthur C. Graesser,et al.  Emotions during learning: The first steps toward an affect sensitive intelligent tutoring system , 2004 .

[27]  Arthur C. Graesser,et al.  Human or Computer? AutoTutor in a Bystander Turing Test , 2002, Intelligent Tutoring Systems.

[28]  Elmar Nöth,et al.  How to find trouble in communication , 2003, Speech Commun..

[29]  Jeremy H. Wright,et al.  Automatically Training a Problematic Dialogue Predictor for a Spoken Dialogue System , 2011, J. Artif. Intell. Res..

[30]  Danielle S. McNamara,et al.  Using LSA in AutoTutor: Learning Through Mixed-Initiative Dialogue in Natural Language , 2007 .

[31]  Giuseppe Riccardi,et al.  How may I help you? , 1997, Speech Commun..

[32]  Diane J. Litman,et al.  Predicting Student Emotions in Computer-Human Tutoring Dialogues , 2004, ACL.

[33]  Amy M. Witherspoon,et al.  Detection of Emotions during Learning with AutoTutor , 2006 .

[34]  Diane J. Litman,et al.  The relative impact of student affect on performance models in a spoken dialogue tutoring system , 2008, User Modeling and User-Adapted Interaction.

[35]  Rosalind W. Picard,et al.  An affective model of interplay between emotions and learning: reengineering educational pedagogy-building a learning companion , 2001, Proceedings IEEE International Conference on Advanced Learning Technologies.

[36]  Elizabeth A. Linnenbrink,et al.  The Role of Motivational Beliefs in Conceptual Change , 2002 .

[37]  M. Csíkszentmihályi.  Flow: The Psychology of Optimal Experience , 1990 .

[38]  Diane J. Litman,et al.  ITSPOKE: An Intelligent Tutoring Spoken Dialogue System , 2004, NAACL.

[39]  A. Graesser,et al.  LEARNING WHILE HOLDING A CONVERSATION WITH A COMPUTER , 2005 .

[40]  Esa Alhoniemi,et al.  Self-Organizing Map for Data Mining in MATLAB: The SOM Toolbox , 1999 .

[41]  Dev Kumar Bose,et al.  Mihalyi Csikszentmihalyi. Flow: The Psychology of Optimal Experience , 2008 .

[42]  R. Azevedo,et al.  Does training on self-regulated learning facilitate students' learning with hypermedia? , 2004 .

[43]  F. Craik,et al.  Memories, Thoughts, and Emotions : Essays in Honor of George Mandler , 1991 .

[44]  John J. B. Allen,et al.  The handbook of emotion elicitation and assessment , 2007 .

[45]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques, 3rd Edition , 1999 .

[46]  Elizabeth A. Linnenbrink,et al.  Role of Affect in Cognitive Processing in Academic Contexts , 2004 .

[47]  Mehryar Mohri,et al.  A comparison of classifiers for detecting emotion from speech , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[48]  Paul Boersma,et al.  Praat: doing phonetics by computer , 2003 .

[49]  Jonathan Klein,et al.  Frustrating the user on purpose: a step toward building an affective computer , 2002, Interact. Comput..

[50]  Arthur C. Graesser,et al.  AutoTutor: an intelligent tutoring system with mixed-initiative dialogue , 2005, IEEE Transactions on Education.

[51]  G. Mandler.  Mind and Body: Psychology of Emotion and Stress , 1984 .

[52]  Nancy L. Stein,et al.  Making sense out of emotion: The representation and use of goal-structured knowledge. , 1990 .

[53]  Albert T. Corbett,et al.  Intelligent Tutoring Systems , 1985, Science.

[54]  M. R. Lepper,et al.  Socializing the intelligent tutor: bringing empathy to computer tutors , 1988 .

[55]  D. Campbell,et al.  Convergent and discriminant validation by the multitrait-multimethod matrix. , 1959, Psychological bulletin.

[56]  A. Schützwohl,et al.  The processing of affectively valenced stimuli: The role of surprise , 2005 .

[57]  Arthur C. Graesser,et al.  Coh-Metrix: Analysis of text on cohesion and language , 2004, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[58]  T. Landauer,et al.  A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge. , 1997 .

[59]  O. G. Selfridge,et al.  Pandemonium: a paradigm for learning , 1988 .

[60]  G. Bower.  Mood and memory. , 1981, The American psychologist.

[61]  Shrikanth S. Narayanan,et al.  Toward detecting emotions in spoken dialogs , 2005, IEEE Transactions on Speech and Audio Processing.

[62]  R. Sternberg,et al.  Motivation, emotion, and cognition: integrative perspectives on intellectual functioning and development , 2004 .

[63]  Robert Kozma,et al.  Chaotic Resonance - Methods and Applications for Robust Classification of noisy and Variable Patterns , 2001, Int. J. Bifurc. Chaos.

[64]  Marianne Miserandino,et al.  Children Who Do Well in School: Individual Differences in Perceived Competence and Autonomy in Above-Average Children , 1996 .

[65]  Arthur C. Graesser,et al.  Deeper Natural Language Processing for Evaluating Student Answers in Intelligent Tutoring Systems , 2006, AAAI.

[66]  Arthur C. Graesser,et al.  Teaching Tactics and Dialog in AutoTutor , 2001 .

[67]  Arthur C. Graesser,et al.  Utterance Classification in AutoTutor , 2003, HLT-NAACL 2003.

[68]  Christine L. Lisetti,et al.  Modeling Multimodal Expression of User’s Affective Subjective Experience , 2002, User Modeling and User-Adapted Interaction.

[69]  James C. Lester,et al.  Modeling self-efficacy in intelligent tutoring systems: An inductive approach , 2008, User Modeling and User-Adapted Interaction.

[70]  Helen Pain,et al.  Informing the Detection of the Students' Motivational State: An Empirical Study , 2002, Intelligent Tutoring Systems.

[71]  Vincent Aleven,et al.  An effective metacognitive strategy: learning by doing and explaining with a computer-based Cognitive Tutor , 2002, Cogn. Sci..

[72]  Mohammed Yeasin,et al.  Robust Recognition of Emotion from Speech , 2006, IVA.

[73]  A. S. J. Botenga,et al.  Results of the Present Study , 1970 .

[74]  Sandra Carberry,et al.  Toward recognizing and conveying an attitude of doubt via natural language , 2002, Appl. Artif. Intell..

[75]  J. Cohn,et al.  Use of Automated Facial Image Analysis for Measurement of Emotion Expression , 2004 .

[76]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[77]  J. Russell.  Core affect and the psychological construction of emotion. , 2003, Psychological review.

[78]  Diane J. Litman,et al.  Predicting Emotion in Spoken Dialogue from Multiple Knowledge Sources , 2004, NAACL.

[79]  Susanne P. Lajoie,et al.  Sherlock: A Coached Practice Environment for an Electronics Troubleshooting Job. , 1988 .

[80]  Carlos Hitoshi Morimoto,et al.  Pupil detection and tracking using multiple light sources , 2000, Image Vis. Comput..

[81]  Arthur C. Graesser,et al.  Predicting Affective States expressed through an Emote-Aloud Procedure from AutoTutor's Mixed-Initiative Dialogue , 2006, Int. J. Artif. Intell. Educ..

[82]  Heng Tao Shen,et al.  Dimensionality Reduction , 2009, Encyclopedia of Database Systems.

[83]  Andreas Stolcke,et al.  Prosody-based automatic detection of annoyance and frustration in human-computer dialog , 2002, INTERSPEECH.

[84]  Cristina Conati,et al.  Probabilistic assessment of user's emotions in educational games , 2002, Appl. Artif. Intell..

[85]  Yanghee Kim,et al.  Empathetic virtual peers enhanced learner interest and self-efficacy , 2005 .

[86]  Dilek Z. Hakkani-Tür,et al.  Using context to improve emotion detection in spoken dialog systems , 2005, INTERSPEECH.

[87]  Alex Pentland,et al.  LAFTER: lips and face real time tracker , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[88]  Michael D. McNeese,et al.  Assessment of User Affective and Belief States for Interface Adaptation: Application to an Air Force Pilot Task , 2002, User Modeling and User-Adapted Interaction.

[89]  J. Bruner.  The act of discovery. , 1961 .

[90]  Yukihiro Matsubara,et al.  Motivation System and Human Model for Intelligent Tutoring , 1996, Intelligent Tutoring Systems.

[91]  K. A. Ericsson,et al.  Protocol Analysis: Verbal Reports as Data , 1984 .

[92]  Min Cheol Whang,et al.  Preparing Computers for Affective Communication: A Psychophysiological Concept and Preliminary Results , 2003, Hum. Factors.

[93]  Shrikanth S. Narayanan,et al.  Combining categorical and primitives-based emotion recognition , 2006, 2006 14th European Signal Processing Conference.

[94]  Manolis Mavrikis,et al.  Diagnosing and acting on student affect: the tutor’s perspective , 2008, User Modeling and User-Adapted Interaction.

[95]  Jennifer Healey,et al.  Toward Machine Emotional Intelligence: Analysis of Affective Physiological State , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[96]  Rosalind W. Picard,et al.  Automated Posture Analysis for Detecting Learner's Interest Level , 2003, 2003 Conference on Computer Vision and Pattern Recognition Workshop.

[97]  K. VanLehn.  Mind Bugs: The Origins of Procedural Misconceptions , 1990 .

[98]  Nilanjan Sarkar,et al.  Affect-sensitive human-robot cooperation - theory and experiments , 2003, 2003 IEEE International Conference on Robotics and Automation (Cat. No.03CH37422).

[99]  Carolyn Penstein Rosé,et al.  The Architecture of Why2-Atlas: A Coach for Qualitative Physics Essay Writing , 2002, Intelligent Tutoring Systems.

[100]  E. Vesterinen,et al.  Affective Computing , 2009, Encyclopedia of Biometrics.

[101]  Michael D. Lee,et al.  A Hierarchical Bayesian Model of Human Decision-Making on an Optimal Stopping Problem , 2006, Cogn. Sci..

[102]  Robert D. Tennyson,et al.  Automating instructional design : computer-based development and delivery tools , 1995 .

[103]  Mehryar Mohri,et al.  Voice signatures , 2003, 2003 IEEE Workshop on Automatic Speech Recognition and Understanding (IEEE Cat. No.03EX721).

[104]  Arthur C. Graesser,et al.  When Are Tutorial Dialogues More Effective Than Reading? , 2007, Cogn. Sci..

[105]  Jonathan Klein,et al.  This computer responds to user frustration: Theory, design, and results , 2002, Interact. Comput..

[106]  Arthur C. Graesser,et al.  Intelligent Tutoring Systems with Conversational Dialogue , 2001, AI Mag..

[107]  Andrew Ortony,et al.  The Cognitive Structure of Emotions , 1988 .

[108]  J. Chase.  Research Overview , 2008 .