Results of the French Evalda-Media evaluation campaign for literal understanding

The aim of the MEDIA-EVALDA project is to evaluate the understanding capabilities of dialog systems. This paper presents the MEDIA protocol for speech understanding evaluation and describes the results of the June 2005 literal evaluation campaign. Five systems, both symbolic or corpus-based participated to the evaluation which is based on a common semantic representation. Different scorings have been performed on the system results. The understanding error rate, for the Full scoring is, depending on the systems, from 29% to 41.3%. A diagnosis analysis of these results is proposed.

[1]  Jonathan G. Fiscus,et al.  A post-processing system to yield reduced word error rates: Recognizer Output Voting Error Reduction (ROVER) , 1997, 1997 IEEE Workshop on Automatic Speech Recognition and Understanding Proceedings.

[2]  Jean-Yves Antoine,et al.  Logical Approach to Natural Language Understanding in a Spoken Dialogue System , 2004, TSD.

[3]  Frédéric Béchet,et al.  On the use of finite state transducers for semantic interpretation , 2006, Speech Commun..

[4]  Matthieu Quignard,et al.  A Deep-Parsing Approach to Natural Language Understanding in Dialogue System: Results of a Corpus-Based Evaluation , 2006, LREC.

[5]  Benoît Crabbé,et al.  Une plate-forme de conception et d’exploitation d’une grammaire d’arbres adjoints lexicalisés , 2003, JEPTALNRECITAL.

[6]  Fernando Pereira,et al.  Weighted finite-state transducers in speech recognition , 2002, Comput. Speech Lang..

[7]  H. Bonneau-Maynard,et al.  A 2+1-level stochastic understanding model , 2005, IEEE Workshop on Automatic Speech Recognition and Understanding, 2005..

[8]  Marilyn A. Walker,et al.  Quantitative and Qualitative Evaluation of Darpa Communicator Spoken Dialogue Systems , 2001, ACL.

[9]  Roger K. Moore.  Computer Speech and Language , 1986 .

[10]  Lynette Hirschman,et al.  Multi-Site Data Collection for a Spoken Language Corpus , 1992, HLT.

[11]  Laila Dybkjær,et al.  The disc approach to spoken language systems development and evaluation , 1998 .

[12]  D. Tribout,et al.  Multi-level information and automatic dialog act detection in human-human spoken dialogs , 2005, Speech Commun..

[13]  Hélène Bonneau-Maynard,et al.  Issues in the development of a stochastic speech understanding system , 2002, INTERSPEECH.