Computer-aided analysis and design for spoken dialogue systems based on quantitative simulations

In this paper, a complete development of computer-aided analysis and design approaches for spoken dialogue systems based on quantitative simulations is presented. With this approach the various performance metrics of a dialogue system can be flexibly defined and numerically evaluated, such that the behavior and performance of the dialogue system can be well predicted and efficiently analyzed before the implementation of the real spoken dialogue system is completed. How the different dialogue performance measures vary with respect to each of the many very complicated factors, regardless of whether it is caused by an individual component, by the overall system design, or by users' response patterns, can be separately identified, because all such factors can he precisely controlled in the simulation. Several analysis examples are presented to show how the approach can be used, including selection and tuning of the speech understanding front end, system strategy design considering query factors and confirmation factors, and objective estimates of user's degree of satisfaction. This approach is therefore very useful for the analysis and design of spoken dialogue systems, although the online test, corpus-based analysis and user survey can always follow after the system is online.

[1]  E. Russell Ritenour,et al.  Evaluating spoken dialog systems for telecommunication services , 1997, EUROSPEECH.

[2]  Jean-Luc Gauvain,et al.  User evaluation of the MASK kiosk , 1998, Speech Commun..

[3]  Lori Lamel,et al.  Data collection for the MASK kiosk: WOz vs. prototype system , 1996, Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96.

[4]  Lori Lamel,et al.  Design strategies for spoken language dialog systems , 1999, 6th European Conference on Speech Communication and Technology (Eurospeech 1999).

[5]  Lou Boves,et al.  Dialogue management in the dutch ARISE train timetable information system , 1999, EUROSPEECH.

[6]  Günther Görz,et al.  Towards understanding spontaneous speech: word accuracy vs. concept accuracy , 1996, Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96.

[7]  Konrad Scheffler,et al.  Probabilistic simulation of human-machine dialogues , 2000, 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.00CH37100).

[8]  Hermann Ney,et al.  Evaluating dialog systems used in the real world , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[9]  Masahiro Araki,et al.  Automatic Evaluation Environment for Spoken Dialogue Systems , 1996, ECAI Workshop on Dialogue Processing in Spoken Language Systems.

[10]  Takahiro Watanabe,et al.  Evaluating Dialogue Strategies under Communication Errors Using Computer-to-Computer Simulation , 1998 .

[11]  Léon J. M. Rothkrantz,et al.  An appreciation study of an ASR inquiry system , 1997, EUROSPEECH.

[12]  Niels Ole Bernsen,et al.  Designing Interactive Speech Systems , 1998, Springer London.

[13]  Hélène Bonneau-Maynard,et al.  Evaluation of dialog strategies for a tourist information retrieval system , 1998, ICSLP.

[14]  Marilyn A. Walker,et al.  Evaluating spoken dialogue agents with PARADISE: Two case studies , 1998, Comput. Speech Lang..

[15]  Gerhard Hanrieder,et al.  Fly with the EAGLES: evaluation of the "ACCeSS" spoken language dialogue system , 1998, ICSLP.

[16]  Niels Ole Bernsen,et al.  Designing interactive speech systems - from first ideas to user testing , 1998 .

[17]  Roberto Pieraccini,et al.  User modeling for spoken dialogue system evaluation , 1997, 1997 IEEE Workshop on Automatic Speech Recognition and Understanding Proceedings.

[18]  Yasuhisa Niimi,et al.  Analysis of interactive strategy to recover from misrecognition of utterances including multiple information items , 1997, EUROSPEECH.

[19]  James R. Glass,et al.  Telephone data collection using the World Wide Web , 1996, Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96.

[20]  Yasuhisa Niimi,et al.  Mathematical analysis of dialogue control strategies , 1999, EUROSPEECH.