Annotating Disengagement for Spoken Dialogue Computer Tutoring

This chapter presents a scheme for annotating student disengagement and its source in a spoken dialogue computer tutoring corpus. The larger goal of this research is to enhance an existing uncertainty-adaptive computer tutor so that it also adapts to disengagement. Our scheme draws on prior work from a wide variety of human–computer interaction domains. Though based on observations of student behavior in our data, our scheme should generalize to other domains. An interannotator agreement study shows that our Disengagement (0.55 Kappa) and Source (0.43 Kappa) labels can be annotated with moderate reliability on par with that of prior work. We conclude by discussing how our Source labels can be used to automatically detect and adapt to disengagement.

[1]  Arthur C. Graesser,et al.  A Time for Emoting: When Affect-Sensitivity Is and Isn't Effective at Promoting Deep Learning , 2010, Intelligent Tutoring Systems.

[2]  Brady Clark,et al.  Responding to Student Uncertainty in Spoken Tutorial Dialogue Systems , 2006, Int. J. Artif. Intell. Educ..

[3]  Ashish Kapoor,et al.  Multimodal affect recognition in learning environments , 2005, ACM Multimedia.

[4]  Diane J. Litman,et al.  Designing and evaluating a wizarded uncertainty-adaptive spoken dialogue tutoring system , 2011, Comput. Speech Lang..

[5]  Ryan Shaun Joazeiro de Baker,et al.  Developing a generalizable detector of when students game the system , 2008, User Modeling and User-Adapted Interaction.

[6]  Georgios N. Yannakakis,et al.  Entertainment capture through heart rate activity in physical interactive playgrounds , 2008, User Modeling and User-Adapted Interaction.

[7]  Diane J. Litman,et al.  Reflection and learning robustness in a natural language conceptual physics tutoring system , 2010 .

[8]  Arthur C. Graesser,et al.  Automatic detection of learner’s affect from conversational cues , 2008, User Modeling and User-Adapted Interaction.

[9]  Ryan Shaun Joazeiro de Baker,et al.  Adapting to When Students Game an Intelligent Tutoring System , 2006, Intelligent Tutoring Systems.

[10]  S. Marsella,et al.  Assessing the validity of appraisal-based models of emotion , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[11]  James C. Lester,et al.  Modeling self-efficacy in intelligent tutoring systems: An inductive approach , 2008, User Modeling and User-Adapted Interaction.

[12]  Helen Pain,et al.  Informing the Detection of the Students' Motivational State: An Empirical Study , 2002, Intelligent Tutoring Systems.

[13]  Eric Horvitz,et al.  Models for Multiparty Engagement in Open-World Dialog , 2009, SIGDIAL Conference.

[14]  Cristina Conati,et al.  Empirically building and evaluating a probabilistic model of user affect , 2009, User Modeling and User-Adapted Interaction.

[15]  Ning Wang,et al.  The politeness effect: Pedagogical agents and learning outcomes , 2008, Int. J. Hum. Comput. Stud..

[16]  Arnon Hershkovitz,et al.  Developing a Log-based Motivation Measuring Tool , 2008, EDM.

[17]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[18]  Joseph E. Beck,et al.  Engagement tracing: using response times to model student disengagement , 2005, AIED.

[19]  Reinhard Pekrun,et al.  Boredom in achievement settings: Exploring control–value antecedents and performance outcomes of a neglected emotion. , 2010 .

[20]  Manolis Mavrikis,et al.  Diagnosing and acting on student affect: the tutor’s perspective , 2008, User Modeling and User-Adapted Interaction.

[21]  Nigel Ward,et al.  Responding to subtle, fleeting changes in the user's internal state , 2001, CHI.

[22]  Carolyn Penstein Rosé,et al.  The Architecture of Why2-Atlas: A Coach for Qualitative Physics Essay Writing , 2002, Intelligent Tutoring Systems.

[23]  Candace L. Sidner,et al.  An Architecture for Engagement in Collaborative Conversations between a Robot and Humans , 2003 .

[24]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[25]  Ma. Mercedes T. Rodrigo,et al.  Comparing the Incidence and Persistence of Learners’ Affect During Interactions with Different Educational Software Packages , 2011 .

[26]  Maria Georgescul,et al.  Proceedings of SIGDIAL , 2006 .

[27]  Diane J. Litman,et al.  The relative impact of student affect on performance models in a spoken dialogue tutoring system , 2008, User Modeling and User-Adapted Interaction.

[28]  Rosalind W. Picard,et al.  An affective model of interplay between emotions and learning: reengineering educational pedagogy-building a learning companion , 2001, Proceedings IEEE International Conference on Advanced Learning Technologies.

[29]  Beverly Park Woolf,et al.  Repairing Disengagement With Non-Invasive Interventions , 2007, AIED.

[30]  Diane J. Litman,et al.  Benefits and challenges of real-time uncertainty detection and adaptation in a spoken dialogue computer tutor , 2011, Speech Commun..

[31]  Sidney K. D'Mello,et al.  What Are You Feeling? Investigating Student Affective States During Expert Human Tutoring Sessions , 2008, Intelligent Tutoring Systems.

[32]  James C. Lester,et al.  Affective Transitions in Narrative-Centered Learning Environments , 2008, J. Educ. Technol. Soc..

[33]  S. Marsella,et al.  Fight the Way You Train:The Role and Limits of Emotions in Training for Combat , 2003 .

[34]  P. Robinson,et al.  Natural Affect Data: Collection and Annotation , 2011 .

[35]  Scotty D. Craig,et al.  Affect and learning: An exploratory look into the role of affect in learning with AutoTutor , 2004 .

[36]  Jack Mostow,et al.  Experimentally augmenting an intelligent tutoring system with human-supplied capabilities: adding human-provided emotional scaffolding to an automated reading tutor that listens , 2002, Proceedings. Fourth IEEE International Conference on Multimodal Interfaces.

[37]  Nicole Novielli,et al.  Attitude Display in Dialogue Patterns , 2008 .