Modelling User Satisfaction and Student Learning in a Spoken Dialogue Tutoring System with Generic, Tutoring, and User Affect Parameters

We investigate using the PARADISE framework to develop predictive models of system performance in our spoken dialogue tutoring system. We represent performance with two metrics: user satisfaction and student learning. We train and test predictive models of these metrics in our tutoring system corpora. We predict user satisfaction with 2 parameter types: 1) system-generic, and 2) tutoring-specific. To predict student learning, we also use a third type: 3) user affect. Although generic parameters are useful predictors of user satisfaction in other PARADISE applications, overall our parameters produce less useful user satisfaction models in our system. However, generic and tutoring-specific parameters do produce useful models of student learning in our system. User affect parameters can increase the usefulness of these models.

[1]  Albert T. Corbett,et al.  Intelligent Tutoring Systems , 1985, Science.

[2]  Elizabeth Shriberg,et al.  Human-Machine Problem Solving Using Spoken Language Systems (SLS): Factors Affecting Performance and User Satisfaction , 1992, HLT.

[3]  Marilyn A. Walker,et al.  PARADISE: A Framework for Evaluating Spoken Dialogue Agents , 1997, ACL.

[4]  Sophie Rosset,et al.  Predictive Performance of Dialog Systems , 2000, LREC.

[5]  Marilyn A. Walker,et al.  Towards developing general models of usability with PARADISE , 2000, Natural Language Engineering.

[6]  Takashi Yamauchi,et al.  Learning from human tutoring , 2001, Cogn. Sci..

[7]  Jack Mostow,et al.  Evaluating tutors that listen: an overview of project LISTEN , 2001 .

[8]  Carolyn Penstein Rosé,et al.  The Architecture of Why2-Atlas: A Coach for Qualitative Physics Essay Writing , 2002, Intelligent Tutoring Systems.

[9]  Gregory A. Sanders,et al.  DARPA communicator: cross-system results for the 2001 evaluation , 2002, INTERSPEECH.

[10]  Andreas Stolcke,et al.  Prosody-based automatic detection of annoyance and frustration in human-computer dialog , 2002, INTERSPEECH.

[11]  Shrikanth S. Narayanan,et al.  Combining acoustic and language information for emotion recognition , 2002, INTERSPEECH.

[12]  Elmar Nöth,et al.  How to find trouble in communication , 2003, Speech Commun..

[13]  Diane J. Litman,et al.  Predicting Student Emotions in Computer-Human Tutoring Dialogues , 2004, ACL.

[14]  Brady Clark,et al.  Evaluating the Effectiveness of SCoT: A Spoken Conversational Tutor , 2004 .

[15]  S. Argamon,et al.  Hedged Responses and Expressions of Affect in Human/Human and Human/Computer Tutorial Interactions , 2004 .

[16]  Scotty D. Craig,et al.  Affect and learning: An exploratory look into the role of affect in learning with AutoTutor , 2004 .

[17]  Claus Zinn,et al.  Generating Tutorial Feedback with Affect , 2004, FLAIRS.

[18]  Diane J. Litman,et al.  Annotating Student Emotional States in Spoken Tutoring Dialogues , 2004, SIGDIAL Workshop.

[19]  Sebastian Möller Towards generic quality prediction models for spoken dialogue systems - a case study , 2005, INTERSPEECH.

[20]  Diane J. Litman,et al.  Correlating student acoustic-prosodic profiles with student learning in spoken tutoring dialogues , 2005, INTERSPEECH.

[21]  Sebastian Möller,et al.  Parameters for Quantifying the Interaction with Spoken Dialogue Telephone Services , 2005, SIGDIAL.

[22]  Joel R. Tetreault,et al.  Comparing Synthesized versus Pre-Recorded Tutor Speech in an Intelligent Tutoring Spoken Dialogue System , 2006, FLAIRS.

[23]  Carolyn Penstein Rosé,et al.  Spoken Versus Typed Human and Computer Dialogue Tutoring , 2006, Int. J. Artif. Intell. Educ..