论文信息 - NIST Language Technology Evaluation Cookbook

NIST Language Technology Evaluation Cookbook

We review some of the methodology applied to the various NIST language technology evaluations. We discuss the elements included in each evaluation plan, and suggest what we believe are key practices for successful evaluations, and what pitfalls should be avoided. A couple of lessons learned are noted.

[1] Ellen M. Voorhees,et al. The TREC Spoken Document Retrieval Track: A Success Story , 2000, TREC.

[2] Jonathan G. Fiscus. Results of the 2003 Topic Detection and Tracking Evaluation , 2004, LREC.

[3] Alvin F. Martin,et al. NIST 2003 language recognition evaluation , 2003, INTERSPEECH.

[4] George R. Doddington,et al. Automatic Evaluation of Machine Translation Quality Using N-gram Co-Occurrence Statistics , 2002 .

[5] Mark A. Przybocki,et al. The Automatic Content Extraction (ACE) Program – Tasks, Data, and Evaluation , 2004, LREC.

[6] David Pallett,et al. A look at NIST'S benchmark ASR tests: past, present, and future , 2003, 2003 IEEE Workshop on Automatic Speech Recognition and Understanding (IEEE Cat. No.03EX721).

[7] Alvin F. Martin,et al. The NIST Speaker Recognition Evaluations: 1996-2001 , 1998, Odyssey.

[8] Alvin F. Martin,et al. The DET curve in assessment of detection task performance , 1997, EUROSPEECH.

[9] Martial Michel,et al. The NIST Meeting Room Pilot Corpus , 2004, LREC.