Emote aloud during learning with AutoTutor: Applying the Facial Action Coding System to cognitive–affective states during learning

In an attempt to discover the facial action units for affective states that occur during complex learning, this study adopted an emote-aloud procedure in which participants were recorded as they verbalised their affective states while interacting with an intelligent tutoring system (AutoTutor). Participants’ facial expressions were coded by two expert raters using Ekman's Facial Action Coding System and analysed using association rule mining techniques. The two expert raters received an overall kappa that ranged between .76 and .84. The association rule mining analysis uncovered facial actions associated with confusion, frustration, and boredom. We discuss these rules and the prospects of enhancing AutoTutor with non-intrusive affect-sensitive capabilities.

[1]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[2]  Elise Ann Earthman.  Creating the Virtual Work: Readers' Processes in Understanding Literary Texts. , 1992 .

[3]  Arthur C. Graesser,et al.  AutoTutor: an intelligent tutoring system with mixed-initiative dialogue , 2005, IEEE Transactions on Education.

[4]  Arthur C. Graesser,et al.  Toward an Affect-Sensitive AutoTutor , 2007, IEEE Intelligent Systems.

[5]  P. Ekman.  Emotions Revealed: Recognizing Faces and Feelings to Improve Communication and Emotional Life , 2003 .

[6]  John J. B. Allen,et al.  The handbook of emotion elicitation and assessment , 2007 .

[7]  Deborah J. Stipek,et al.  Motivation to Learn: From Theory to Practice , 1992 .

[8]  K. A. Ericsson,et al.  Protocol Analysis: Verbal Reports as Data , 1984 .

[9]  Andrew Ortony,et al.  The Cognitive Structure of Emotions , 1988 .

[10]  T. Goetz,et al.  Academic Emotions in Students' Self-Regulated Learning and Achievement: A Program of Qualitative and Quantitative Research , 2002 .

[11]  Joseph P. Magliano,et al.  Conscious understanding during comprehension , 1996 .

[12]  Scotty D. Craig,et al.  Affect and learning: An exploratory look into the role of affect in learning with AutoTutor , 2004 .

[13]  Matthew W. Lewis,et al.  Self-Explonations: How Students Study and Use Examples in Learning to Solve Problems , 1989, Cogn. Sci..

[14]  Amy L. Eva-Wood.  How Think-and-Feel-Aloud Instruction Influences Poetry Readers , 2004 .

[15]  Rosalind W. Picard,et al.  Towards a Learning Companion that Recognizes Affect , 2001 .

[16]  J. Eccles.  The emotional brain. , 1980, Bulletin et memoires de l'Academie royale de medecine de Belgique.

[17]  T. Dalgleish,et al.  Handbook of cognition and emotion , 1999 .

[18]  Allen Newell,et al.  Human Problem Solving. , 1973 .

[19]  Diane J. Litman,et al.  Correlations between dialogue acts and learning in spoken tutoring dialogues , 2006, Natural Language Engineering.

[20]  A. Graesser,et al.  PREG: Elements of a Model of Question Asking , 2001 .

[21]  M. Lepper,et al.  Motivational techniques of expert human tutors: Lessons for the design of computer-based tutors. , 1993 .

[22]  Arthur C. Graesser,et al.  Question asking and eye tracking during cognitive disequilibrium: Comprehending illustrated texts on devices when the devices break down , 2005, Memory & cognition.

[23]  Ramakrishnan Srikant,et al.  Fast Algorithms for Mining Association Rules in Large Databases , 1994, VLDB.

[24]  Sean A. Spence,et al.  Descartes' Error: Emotion, Reason and the Human Brain , 1995 .

[25]  Maja Pantic,et al.  Expert system for automatic analysis of facial expressions , 2000, Image Vis. Comput..

[26]  J. Cohn,et al.  Use of Automated Facial Image Analysis for Measurement of Emotion Expression , 2004 .

[27]  Gary E. Raney,et al.  An eye movement study of insight problem solving , 2001, Memory & cognition.

[28]  E. Vesterinen,et al.  Affective Computing , 2009, Encyclopedia of Biometrics.

[29]  Takashi Yamauchi,et al.  Learning from human tutoring , 2001, Cogn. Sci..

[30]  Cristina Conati,et al.  Probabilistic assessment of user's emotions in educational games , 2002, Appl. Artif. Intell..

[31]  L. F. Barrett.  Are Emotions Natural Kinds? , 2006, Perspectives on psychological science : a journal of the Association for Psychological Science.

[32]  Debra K. Meyer,et al.  Discovering Emotion in Classroom Motivation Research , 2002 .

[33]  P. Rozin,et al.  High frequency of facial expressions corresponding to confusion, concentration, and worry in an analysis of naturally occurring facial expressions of Americans. , 2003, Emotion.