Question Ranking and Selection in Tutorial Dialogues

A key challenge for dialogue-based intelligent tutoring systems lies in selecting follow-up questions that are not only context relevant but also encourage self-expression and stimulate learning. This paper presents an approach to ranking candidate questions for a given dialogue context and introduces an evaluation framework for this task. We learn to rank using judgments collected from expert human tutors, and we show that adding features derived from a rich, multi-layer dialogue act representation improves system performance over baseline lexical and syntactic features to a level in agreement with the judges. The experimental results highlight the important factors in modeling the questioning process. This work provides a framework for future work in automatic question generation and it represents a step toward the larger goal of directly learning tutorial dialogue policies directly from human examples.

[1]  Kristy Elizabeth Boyer,et al.  An Empirically-Derived Question Taxonomy for Task-Oriented Tutorial Dialogue , 2009 .

[2]  Johanna D. Moore,et al.  A comparative evaluation of socratic versus didactic tutoring , 2001 .

[3]  Maxine Eskénazi,et al.  Predicting Change in Student Motivation by Measuring Cohesion between Tutor and Student , 2011, BEA@ACL.

[4]  Kristy Elizabeth Boyer,et al.  Discovering Tutorial Dialogue Strategies with Hidden Markov Models , 2009, AIED.

[5]  C. Rosé.  The Role of Why Questions in Effective Human Tutoring , 2003 .

[6]  Yoram Singer,et al.  Learning to Order Things , 1997, NIPS.

[7]  A. Graesser,et al.  Question Asking During Tutoring , 1994 .

[8]  Manish Agarwal,et al.  Automatic Gap-fill Question Generation from Text Books , 2011, BEA@ACL.

[9]  Mark G. Core,et al.  Coding Dialogs with the DAMSL Annotation Scheme , 1997 .

[10]  Magdalena Wolska,et al.  A Classification of Dialogue Actions in Tutorial Dialogue , 2008, COLING.

[11]  Dimitra Tsovaltzi,et al.  Formalising Hinting in Tutorial Dialogues , 2001 .

[12]  Reva Freedman,et al.  Learning the Use of Discourse Markers in Tutorial Dialogue for an Intelligent Tutoring System , 2000 .

[13]  Kurt VanLehn,et al.  To Elicit Or To Tell: Does It Matter? , 2009, AIED.

[14]  Noah A. Smith,et al.  Good Question! Statistical Ranking for Question Generation , 2010, NAACL.

[15]  Ronald A. Cole,et al.  My science tutor: A conversational multimedia virtual tutor for elementary school science , 2011, TSLP.

[16]  Thorsten Joachims,et al.  Making large scale SVM learning practical , 1998 .

[17]  Xuchen Yao,et al.  Question Generation with Minimal Recursion Semantics , 2010 .

[18]  Arthur C. Graesser,et al.  When Are Tutorial Dialogues More Effective Than Reading? , 2007, Cogn. Sci..

[19]  R. Catrambone,et al.  Proceedings of the 32nd Annual Conference of the Cognitive Science Society , 2010 .

[21]  Lee BECKER,et al.  What a pilot study says about running a question generation challenge , 2009 .

[22]  Wayne H. Ward.  Extracting information in spontaneous speech , 1994, ICSLP.

[23]  Sarel van Vuuren,et al.  DISCUSS: A dialogue move taxonomy layered over semantic representations , 2011, IWCS.

[24]  William C. Mann,et al.  Rhetorical Structure Theory: Description and Construction of Text Structures , 1987 .

[25]  Margaret G. McKeown,et al.  Questioning the Author: A Yearlong Classroom Implementation to Engage Students with Text , 1996, The Elementary School Journal.

[26]  Marilyn A. Walker,et al.  SPoT: A Trainable Sentence Planner , 2001, NAACL.

[27]  Marilyn A. Walker,et al.  Evaluating a Trainable Sentence Planner for a Spoken Dialogue System , 2001, ACL.

[28]  Harry Bunt,et al.  The DIT++ taxanomy for functional dialogue markup , 2009 .

[29]  Kurt VanLehn,et al.  Reinforcement Learning-based Feature Seleciton For Developing Pedagogically Effective Tutorial Dialogue Tactics , 2008, EDM.

[30]  Arthur C. Graesser,et al.  AutoTutor: an intelligent tutoring system with mixed-initiative dialogue , 2005, IEEE Transactions on Education.

[31]  Jean Carletta,et al.  Assessing Agreement on Classification Tasks: The Kappa Statistic , 1996, CL.

[32]  Diane J. Litman,et al.  ITSPOKE: An Intelligent Tutoring Spoken Dialogue System , 2004, NAACL.

[33]  Dimitra Tsovaltzi,et al.  A View on Dialogue Move Taxonomies for Tutorial Dialogues , 2004, SIGDIAL Workshop.

[34]  Kurt VanLehn,et al.  Do Micro-Level Tutorial Decisions Matter: Applying Reinforcement Learning to Induce Pedagogical Tutorial Tactics , 2010, Intelligent Tutoring Systems.