Improving interpretation robustness in a tutorial dialogue system

We present an experiment aimed at improving interpretation robustness of a tutorial dialogue system that relies on detailed semantic interpretation and dynamic natural language feedback generation. We show that we can improve overall interpretation quality by combining the output of a semantic interpreter with that of a statistical classifier trained on the subset of student utterances where semantic interpretation fails. This improves on a previous result which used a similar approach but trained the classifier on a substantially larger data set containing all student utterances. Finally, we discuss how the labels from the statistical classifier can be integrated effectively with the dialogue system’s existing error recovery policies.

[1]  Alexander S. Yeh,et al.  More accurate tests for the statistical significance of result differences , 2000, COLING.

[2]  Diane J. Litman,et al.  ITSPOKE: An Intelligent Tutoring Spoken Dialogue System , 2004, NAACL.

[3]  Ramzan Khuwaja,et al.  Architecture of CIRCSIM-Tutor (v.3): a smart cardiovascular physiology tutor , 1994, Proceedings of IEEE Symposium on Computer-Based Medical Systems (CBMS).

[4]  Rodney D. Nielsen,et al.  Learning to Assess Low-Level Conceptual Understanding , 2008, FLAIRS Conference.

[5]  Alexander I. Rudnicky,et al.  Sorry and I Didn’t Catch That! - An Investigation of Non-understanding Errors and Recovery Strategies , 2005, SIGDIAL.

[6]  W. Lewis Johnson,et al.  Validation of a Dialog System for Language Learners , 2010, SIGDIAL Conference.

[7]  Raymond J. Mooney,et al.  Learning Synchronous Grammars for Semantic Parsing with Lambda Calculus , 2007, ACL.

[8]  Reva Freedman.  Using a Reactive Planner as the Basis for a Dialogue Agent , 2000, FLAIRS Conference.

[9]  Pamela W. Jordan,et al.  Evidence of Misunderstandings in Tutorial Dialogue and their Impact on Learning , 2009, AIED.

[10]  Johanna D. Moore,et al.  Beetle II: A System for Tutoring and Computational Linguistics Experimentation , 2010, ACL.

[11]  Luke S. Zettlemoyer,et al.  Learning to Map Sentences to Logical Form: Structured Classification with Probabilistic Categorial Grammars , 2005, UAI.

[12]  Mark Steedman,et al.  Inducing Probabilistic CCG Grammars from Logical Form with Higher-Order Unification , 2010, EMNLP.

[13]  Rada Mihalcea,et al.  Learning to Grade Short Answer Questions using Semantic Similarity Measures and Dependency Graph Alignments , 2011, ACL.

[14]  Vincent Aleven,et al.  Pilot-Testing a Tutorial Dialogue System That Supports Self-Explanation , 2002, Intelligent Tutoring Systems.

[15]  Martin Chodorow,et al.  C-rater: Automated Scoring of Short-Answer Questions , 2003, Comput. Humanit..

[16]  Merrilea J. Mayo,et al.  Games for science and engineering education , 2007, CACM.

[17]  Stephen G. Pulman,et al.  Automatic Short Answer Marking , 2005, ACL 2005.

[18]  Arthur C. Graesser,et al.  AutoTutor: A simulation of a human tutor , 1999, Cognitive Systems Research.

[19]  Johanna D. Moore,et al.  The Impact of Interpretation Problems on Tutorial Dialogue , 2010, ACL.

[20]  V. Shute.  Focus on Formative Feedback , 2007 .

[21]  Johanna D. Moore,et al.  Combining Semantic Interpretation and Statistical Classification for Improved Explanation Processing in a Tutorial Dialogue System , 2013, AIED.

[22]  Kurt VanLehn,et al.  A Natural Language Tutorial Dialogue System for Physics , 2006, FLAIRS Conference.

[23]  Chris Brew,et al.  Towards Effective Tutorial Feedback for Explanation Questions: A Dataset and Baselines , 2012, HLT-NAACL.

[24]  Kurt VanLehn,et al.  Developing pedagogically effective tutorial dialogue tactics: experiments and a testbed , 2007, SLaTE.

[25]  Martha Palmer,et al.  Annotating Students' Understanding of Science Concepts , 2008, LREC.

[26]  V. Shute.  Focus on Formative Feedback , 2008 .

[27]  Johanna D. Moore,et al.  Evaluating language understanding accuracy with respect to objective outcomes in a dialogue system , 2012, EACL.

[28]  Jan T. van der Veen,et al.  The learning effects of computer simulations in science education , 2012, Comput. Educ..

[29]  Brady Clark,et al.  Advantages of Spoken Language Interaction in Dialogue-Based Intelligent Tutoring Systems , 2004, Intelligent Tutoring Systems.

[30]  Johanna D. Moore,et al.  Talk Like an Electrician: Student Dialogue Mimicking Behavior in an Intelligent Tutoring System , 2011, AIED.