Identifying Effective Moves in Tutoring: On the Refinement of Dialogue Act Annotation Schemes

The rich natural language dialogue that is exchanged between tutors and students has inspired many successful lines of research on tutorial dialogue systems. Yet, today’s tutorial dialogue systems do not regularly achieve the same level of student learning gain as has been observed with expert human tutors. Implementing models directly informed by, and even machine-learned from, human-human tutorial dialogue is highly promising. With this goal in mind, this paper makes two contributions to tutorial dialogue systems research. First, it presents a dialogue act annotation scheme that is designed specifically to address a common weakness within dialogue act tag sets, namely, their dominance by a single large majority dialogue act class. Second, using this new fine-grained annotation scheme, the paper describes important correlations uncovered between tutor dialogue acts and student learning gain within a corpus of tutorial dialogue for introductory computer science. These findings can inform the design of future tutorial dialogue systems by suggesting ways in which systems can adapt at a fine-grained level to student actions.

[1]  B. Bloom.  The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring , 1984 .

[2]  Susanne P. Lajoie,et al.  Motivational Techniques of Expert Human Tutors: Lessons for the Design of Computer-Based Tutors , 2013 .

[3]  Davide Fossati,et al.  Beyond the code-and-count analysis of tutoring dialogues , 2007, AIED.

[4]  Arthur C. Graesser,et al.  When Are Tutorial Dialogues More Effective Than Reading? , 2007, Cogn. Sci..

[5]  Carolyn Penstein Rosé,et al.  Socially Capable Conversational Tutors Can Be Effective in Collaborative Learning Situations , 2010, Intelligent Tutoring Systems.

[6]  Chen-Lin C. Kulik,et al.  Educational Outcomes of Tutoring: A Meta-analysis of Findings , 1982 .

[7]  Andrew Olney,et al.  Mining Collaborative Patterns in Tutorial Dialogues , 2010, EDM 2010.

[8]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[9]  Mark G. Core,et al.  Coding Dialogs with the DAMSL Annotation Scheme , 1997 .

[10]  Kristy Elizabeth Boyer,et al.  Recognizing Effective and Student-Adaptive Tutor Moves in Task-Oriented Tutorial Dialogue , 2012, FLAIRS Conference.

[11]  Diane J. Litman,et al.  Adapting to Student Uncertainty Improves Tutoring Dialogues , 2009, AIED.

[12]  Joseph P. Magliano,et al.  Collaborative dialogue patterns in naturalistic one-to-one tutoring , 1995 .

[13]  Takashi Yamauchi,et al.  Learning from human tutoring , 2001, Cogn. Sci..

[14]  A. Graesser,et al.  A Motivationally Supportive Affect-Sensitive AutoTutor , 2011 .

[15]  Stefanie N. Lindstaedt,et al.  Sustaining TEL: From Innovation to Learning and Practice , 2010, Lecture Notes in Computer Science.

[16]  John Seely Brown,et al.  Intelligent Tutoring Systems , 2016, Lecture Notes in Computer Science.

[17]  Davide Fossati,et al.  Exploring Effective Dialogue Act Sequences in One-on-one Computer Science Tutoring Dialogues , 2011, BEA@ACL.

[18]  M. Lepper,et al.  Motivational techniques of expert human tutors: Lessons for the design of computer-based tutors. , 1993 .

[19]  Kurt VanLehn,et al.  Do Micro-Level Tutorial Decisions Matter: Applying Reinforcement Learning to Induce Pedagogical Tutorial Tactics , 2010, Intelligent Tutoring Systems.

[20]  Jennifer Chu-Carroll,et al.  A Statistical Model for Discourse Act Recognition in Dialogue Interactions , 1998 .

[21]  Diane J. Litman,et al.  Correlations between dialogue acts and learning in spoken tutoring dialogues , 2006, Natural Language Engineering.

[22]  Johanna D. Moore,et al.  Content, Social, and Metacognitive Statements: An Empirical Study Comparing Human-Human and Human-Computer Tutorial Dialogue , 2010, EC-TEL.