Evaluating discourse understanding in spoken dialogue systems

This article describes a method for creating an evaluation measure for discourse understanding in spoken dialogue systems. No well-established measure has yet been proposed for evaluating discourse understanding, which has made it necessary to evaluate it only on the basis of the system's total performance. Such evaluations, however, are greatly influenced by task domains and dialogue strategies. To find a measure that enables good estimation of system performance only from discourse understanding results, we enumerated possible discourse-understanding-related metrics and calculated their correlation with the system's total performance through dialogue experiments.

[1]  Masanobu Abe,et al.  A Japanese TTS system based on multiform units and a speech modification algorithm with harmonics reconstruction , 2001, IEEE Trans. Speech Audio Process..

[2]  Stephanie Seneff,et al.  Response planning and generation in the MERCURY flight reservation system , 2002, Comput. Speech Lang..

[3]  Daniel G. Bobrow,et al.  A frame driven dialog system , 1980 .

[4]  Norihito Yasuda,et al.  Efficient spoken dialogue control depending on the speech recognition rate and system's database , 2003, INTERSPEECH.

[5]  Kiyohiro Shikano,et al.  Julius - an open source real-time large vocabulary recognition engine , 2001, INTERSPEECH.

[6]  C. Raymond Perrault,et al.  Analyzing Intention in Utterances , 1986, Artif. Intell..

[7]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques with Java implementations , 2002, SGMD.

[8]  Mikio Nakano,et al.  WIT: A Toolkit for Building Robust and Real-Time Spoken Dialogu Systems , 2000, SIGDIAL Workshop.

[9]  Marilyn A. Walker,et al.  PARADISE: A Framework for Evaluating Spoken Dialogue Agents , 1997, ACL.

[10]  Sandra Carberry,et al.  Plan Recognition in Natural Language Dialogue , 1990 .

[11]  Michael F. McTear,et al.  Book Review: Spoken Dialogue Technology: Toward the Conversational User Interface, by Michael F. McTear , 2002, CL.

[12]  Daniel G. Bobrow,et al.  GUS, A Frame-Driven Dialog System , 1986, Artif. Intell..

[13]  Jennifer Chu-Carroll,et al.  MIMIC: An Adaptive Mixed Initiative Spoken Dialogue System for Information Queries , 2000, ANLP.

[14]  Giuseppe Riccardi,et al.  How may I help you? , 1997, Speech Commun..

[15]  James F. Allen,et al.  An architecture for more realistic conversational systems , 2001, IUI '01.

[16]  Bernhard Schölkopf,et al.  A tutorial on support vector regression , 2004, Stat. Comput..

[17]  Victor Zue,et al.  Data collection and performance evaluation of spoken dialogue systems: the MIT experience , 2000, INTERSPEECH.

[18]  Alexander I. Rudnicky,et al.  Task and domain specific modelling in the Carnegie Mellon communicator system , 2000, INTERSPEECH.

[19]  Marilyn A. Walker,et al.  Towards developing general models of usability with PARADISE , 2000, Natural Language Engineering.

[20]  Ian H. Witten,et al.  Induction of model trees for predicting continuous classes , 1996 .

[21]  Lori Lamel,et al.  The LIMSI ARISE system , 2000, Speech Commun..

[22]  Ian Witten,et al.  Data Mining , 2000 .

[23]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[24]  Candace L. Sidner,et al.  COLLAGEN: Applying Collaborative Discourse Theory to Human-Computer Interaction , 2001, AI Mag..

[25]  Yuji Matsumoto,et al.  Extracting Important Sentences with Support Vector Machines , 2002, COLING.

[26]  Mikio Nakano,et al.  A method for evaluating incremental utterance understanding in spoken dialogue systems , 2002, INTERSPEECH.

[27]  Lou Boves,et al.  Dialogue management in the dutch ARISE train timetable information system , 1999, EUROSPEECH.

[28]  Lori Meiskey,et al.  An object-oriented approach to dialogue management in spoken language systems , 1994, CHI Conference Companion.

[29]  Tatsuya Kawahara,et al.  Flexible Mixed-Initiative Dialogue Management using Concept-Level Confidence Measures of Speech Recognizer Output , 2000, COLING.

[30]  Mikio Nakano,et al.  Corpus-Based Discourse Understanding in Spoken Dialogue Systems , 2003, ACL.

[31]  Joseph Polifroni,et al.  A form-based dialogue manager for spoken language applications , 1996, Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96.

[32]  Bob Carpenter,et al.  Vector-based Natural Language Call Routing , 1999, Comput. Linguistics.

[33]  Allen L. Gorin,et al.  Construct Algebra: Analytical Dialog Management , 1999, ACL.

[34]  Victor Zue,et al.  Conversational interfaces: advances and challenges , 1997, Proceedings of the IEEE.

[35]  Lynette Hirschman,et al.  Comparing Several Aspects of Human-Computer and Human-Human Dialogues , 2001, SIGDIAL Workshop.