论文信息 - Evaluating DUC 2005 using Basic Elements

Evaluating DUC 2005 using Basic Elements

In this paper we introduce Basic Elements, a new way of automating the evaluation of text summaries. We show that this method correlates better with human judgments than any other automated procedure to date, and overcomes the subjectivity/variability problems of manual methods that require humans to preprocess summaries to be evaluated. This is demonstrated on DUC 2005 peer systems and peer-produced summaries.

[1] Sadaoki Furui,et al. Evaluation method for automatic speech summarization , 2003, INTERSPEECH.

[2] Eugene Charniak,et al. A Maximum-Entropy-Inspired Parser , 2000, ANLP.

[3] Ani Nenkova,et al. Evaluating Content Selection in Summarization: The Pyramid Method , 2004, NAACL.

[4] Patrick Pantel,et al. Concept Discovery from Text , 2002, COLING.

[5] A Maximum-Entropy-Inspired , 1999 .

[6] Eduard Hovy,et al. Manual and automatic evaluation of summaries , 2002, ACL 2002.

[7] David M. Magerman. Natural Language Parsing as Statistical Pattern Recognition , 1994, ArXiv.

[8] Eduard H. Hovy,et al. Automatic Evaluation of Summaries Using N-gram Co-occurrence Statistics , 2003, NAACL.

[9] Dekang Lin,et al. A dependency-based method for evaluating broad-coverage parsers , 1995, Natural Language Engineering.

[10] Simone Teufel,et al. Examining the consensus between human summaries: initial experiments with factoid analysis , 2003, HLT-NAACL 2003.

[11] Julio Gonzalo,et al. An Empirical Study of Information Synthesis Task , 2004, ACL.

[12] Dragomir R. Radev,et al. Summarization evaluation using relative utility , 2003, CIKM '03.

[13] Michael Collins,et al. Head-Driven Statistical Models for Natural Language Parsing , 2003, CL.

[14] Salim Roukos,et al. Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.