Evaluation of the NLP Components of the OVIS2 Spoken Dialogue System

The NWO Priority Programme Language and Speech Technology is a 5-year research programme aiming at the development of spoken language information systems. In the Programme, two alternative natural language processing (NLP) modules are developed in parallel: a grammarbased (conventional, rule-based) module and a data-oriented (memorybased, stochastic, DOP) module. In order to compare the NLP modules, a formal evaluation has been carried out three years after the start of the Programme. This paper describes the evaluation procedure and the evaluation results. The grammar-based component performs much better than the data-oriented one in this comparison.

[1]  Rens Bod,et al.  Data-Oriented Language Processing. An Overview , 1996, ArXiv.

[2]  Rens Bod.  Using an Annotated Corpus as a Stochastic Grammar , 1993, EACL.

[3]  Khalil Sima'an,et al.  Disambiguation and Interpretation of Wordgraphs using Data-Oriented Parsing , 1996 .

[4]  Ivan A. Sag,et al.  Book Reviews: Head-driven Phrase Structure Grammar and German in Head-driven Phrase-structure Grammar , 1996, CL.

[5]  Remko Scha,et al.  A Corpus-based Approach to Semantic Interpretation , 1994 .

[6]  Khalil Simaan,et al.  Computational Complexity of Probabilistic Disambiguation by means of Tree-Grammars , 1996, COLING.

[7]  Gertjan van Noord,et al.  On the Intersection of Finite State Automata and De nite Clause Grammars , 1994 .

[8]  H. Alshawi,et al.  The Core Language Engine , 1994 .

[9]  Günther Görz,et al.  Towards understanding spontaneous speech: word accuracy vs. concept accuracy , 1996, Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96.

[10]  Gertjan van Noord.  The Intersection of Finite State Automata and Definite Clause Grammars , 1995, ACL.

[11]  NederhofMark-Jan,et al.  Robust grammatical analysis for spoken dialogue systems , 1999 .

[12]  Khalil Sima’an,et al.  An optimised algorithm for data oriented parsing , 1997 .

[13]  Mark-Jan Nederhof,et al.  Robust grammatical analysis for spoken dialogue systems , 1999, Natural Language Engineering.

[14]  Mark-Jan Nederhof,et al.  Conventional Natural Language Processing in the NWO Priority Programme on Language and Speech Technology October 1996 Deliverables , 1996 .

[15]  Remko Scha,et al.  Priority programme 'Language and speech technology' , 1997 .

[16]  Hermann Ney,et al.  Word graphs: an efficient interface between continuous-speech recognition and language understanding , 1993, 1993 IEEE International Conference on Acoustics, Speech, and Signal Processing.