How to Classify Tutorial Dialogue? Comparing Feature Vectors vs. Sequences

A key issue in using machine learning to classify tutorial dialogues is how to represent time-varying data. Standard classifiers take as input a feature vector and output its predicted label. It is possible to formulate tutorial dialogue classification problems in this way. However, a feature vector representation requires mapping a dialogue onto a fixed number of features, and does not innately exploit its sequential nature. In contrast, this paper explores a recent method that classifies sequences, using a technique new to the Educational Data Mining community – Hidden Conditional Random Fields [Quattoni et al., 2007]. We illustrate its application to a data set from Project LISTEN's Reading Tutor, and compare it to three baselines using the same data, crossvalidation splits, and feature set. Our technique produces state-of-the-art classification accuracy in predicting reading task completion. We consider the contributions of this paper to be (i) introducing HCRFs to the EDM community, (ii) formulating tutorial dialogue classification as a sequence classification problem, and (iii) evaluating and comparing dialogue classification.

[1]  L. Breiman.  CONSISTENCY FOR A SIMPLE MODEL OF RANDOM FORESTS , 2004 .

[2]  Kurt VanLehn,et al.  Do Micro-Level Tutorial Decisions Matter: Applying Reinforcement Learning to Induce Pedagogical Tutorial Tactics , 2010, Intelligent Tutoring Systems.

[3]  Sebastian Möller,et al.  Evaluating spoken dialogue systems according to de-facto standards: A case study , 2007, Comput. Speech Lang..

[4]  Marilyn A. Walker,et al.  Reinforcement Learning for Spoken Dialogue Systems , 1999, NIPS.

[5]  Joseph E. Beck,et al.  Engagement tracing: using response times to model student disengagement , 2005, AIED.

[6]  Kurt VanLehn,et al.  Reinforcement Learning-based Feature Seleciton For Developing Pedagogically Effective Tutorial Dialogue Tactics , 2008, EDM.

[7]  Sebastian Möller,et al.  Predicting the quality and usability of spoken dialogue services , 2008, Speech Commun..

[8]  Trevor Darrell,et al.  Hidden Conditional Random Fields , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Charles M. Reigeluth,et al.  Instructional Design Theories and Models : An Overview of Their Current Status , 1983 .

[10]  Alan W. Black,et al.  Describing Spoken Dialogue Systems Differences , 2008 .

[11]  Michael I. Jordan,et al.  Multi-task feature selection , 2006 .

[12]  Nello Cristianini,et al.  Classification using String Kernels , 2000 .

[13]  Marilyn A. Walker,et al.  Towards developing general models of usability with PARADISE , 2000, Natural Language Engineering.

[14]  Mei-Yuh Hwang,et al.  The SPHINX-II speech recognition system: an overview , 1993, Comput. Speech Lang..

[15]  Adam L. Berger,et al.  A Maximum Entropy Approach to Natural Language Processing , 1996, CL.

[16]  Hua Ai,et al.  Comparing User Simulation Models For Dialog Strategy Learning , 2007, HLT-NAACL.

[17]  Joseph E. Beck,et al.  High-Level Student Modeling with Machine Learning , 2000, Intelligent Tutoring Systems.

[18]  H. Zou,et al.  Regularization and variable selection via the elastic net , 2005 .

[19]  Jack Mostow,et al.  Evaluating tutors that listen: an overview of project LISTEN , 2001 .

[20]  Hanna M. Wallach,et al.  Topic modeling: beyond bag-of-words , 2006, ICML.

[21]  Regina Barzilay,et al.  Gestural Cohesion for Topic Segmentation , 2008, ACL.

[22]  Jack Mostow,et al.  Classifying dialogue in high-dimensional space , 2011, TSLP.

[23]  Kevin P. Murphy,et al.  Machine learning - a probabilistic perspective , 2012, Adaptive computation and machine learning series.

[24]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[25]  Trevor Darrell,et al.  Hidden Conditional Random Fields for Gesture Recognition , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[26]  Jérôme Louradour,et al.  SVM Speaker Verification using an Incomplete Cholesky Decomposition Sequence Kernel , 2006, 2006 IEEE Odyssey - The Speaker and Language Recognition Workshop.

[27]  Kristy Elizabeth Boyer,et al.  Characterizing the Effectiveness of Tutorial Dialogue with Hidden Markov Models , 2010, Intelligent Tutoring Systems.

[28]  Melita Hajdinjak,et al.  The PARADISE Evaluation Framework: Issues and Findings , 2006, Computational Linguistics.

[29]  Jian Pei,et al.  A brief survey on sequence classification , 2010, SKDD.

[30]  Alex Acero,et al.  Hidden conditional random fields for phone classification , 2005, INTERSPEECH.

[31]  J M Bland,et al.  Weighted comparison of means , 1998, BMJ.

[32]  Jack Mostow,et al.  Predicting Task Completion from Rich but Scarce Data , 2010, EDM.

[33]  Andreas Stolcke,et al.  Dialogue act modeling for automatic tagging and recognition of conversational speech , 2000, CL.