Considerations in the design and evaluation of spoken language dialog systems

In this paper we summarize our experience at LIMSI in the design, development and evaluation of spoken language dialog systems for information retrieval tasks. This work has been for the most part carried out in the context of several European and international projects. Evaluation plays an integral role in the development of spoken language dialog systems. While there are commonly used measures and methodologies for evaluating speech recognizers, the evaluation of spoken dialog systems is considerably more complicated due to the interactive nature and the human perception of performance. It is therefore important to assess not only the individual system components, but the overall system performance using objective and subjective measures.

[1]  Jean-Luc Gauvain,et al.  User evaluation of the MASK kiosk , 1998, Speech Commun..

[2]  Hélène Bonneau-Maynard,et al.  Evaluation of dialog strategies for a tourist information retrieval system , 1998, ICSLP.

[3]  David Goodine,et al.  A French version of the MIT-ATIS system: portability issues , 1993, EUROSPEECH.

[4]  E. Levin,et al.  CHRONUS, The next generation , 1995 .

[5]  Lori Lamel,et al.  Data collection for the MASK kiosk: WOz vs. prototype system , 1996, Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96.

[6]  Bernhard Rüber,et al.  Context-dependent probability adaptation in speech understanding , 1997, Comput. Speech Lang..

[7]  Jean-Luc Gauvain,et al.  Speech recognition for an information kiosk , 1996, Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96.

[8]  Jean-Luc Gauvain,et al.  The LIMSI RailTel System: Field trial of a telephone service for rail travel information , 1997, Speech Commun..

[9]  James R. Glass,et al.  Real-time telephone-based speech recognition in the Jupiter domain , 1999, 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.99CH36258).

[10]  Hélène Bonneau-Maynard,et al.  A framework for evaluating contextual understanding , 2000, INTERSPEECH.

[11]  Richard M. Schwartz,et al.  Language understanding using hidden understanding models , 1996, Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96.

[12]  Jean-Luc Gauvain,et al.  Spoken Language Component of the MASK Kiosk , 1997 .

[13]  Lori Lamel,et al.  Design strategies for spoken language dialog systems , 1999, EUROSPEECH.

[14]  Lori Lamel,et al.  The LIMSI ARISE system , 2000, Speech Commun..

[15]  Niels Ole Bernsen,et al.  Current practice in the development and evaluation of spoken language dialogue systems , 1999, EUROSPEECH.