DUC in context

Recent years have seen increased interest in text summarization with emphasis on evaluation of prototype systems. Many factors can affect the design of such evaluations, requiring choices among competing alternatives. This paper examines several major themes running through three evaluations: SUMMAC, NTCIR, and DUC, with a concentration on DUC. The themes are extrinsic and intrinsic evaluation, evaluation procedures and methods, generic versus focused summaries, single- and multi-document summaries, length and compression issues, extracts versus abstracts, and issues with genre.

[1]  Sasha Blair-Goldensohn.  From Definitions to Complex Topics: Columbia University at DUC 2005 , 2005 .

[2]  Terry Copeck,et al.  Vocabulary Agreement Among Model Summaries And Source Documents 1 , 2004 .

[3]  Paul Over,et al.  Intrinsic Evaluation of Generic News Text Summarization Systems , 2003 .

[4]  Inderjeet Mani,et al.  The Tipster Summac Text Summarization Evaluation , 1999, EACL.

[5]  Chin-Yew Lin,et al.  ROUGE: A Package for Automatic Evaluation of Summaries , 2004, ACL 2004.

[6]  Zhu Zhang,et al.  NewsInEssence: A System For Domain-Independent, Real-Time News Clustering and Multi-Document Summarization , 2001, HLT.

[7]  Andrew Hickl,et al.  Lite-GISTexter at DUC 2005 , 2005 .

[8]  Vasudeva Varma,et al.  A Relevance-Based Language Modeling approach to DUC 2005 , 2005 .

[9]  Ani Nenkova,et al.  Evaluating Content Selection in Summarization: The Pyramid Method , 2004, NAACL.

[10]  M. Sanderson.  Advances in Automatic Text Summarization edited by Inderjeet Mani and Mark T. Maybury , 2000 .

[11]  B. Magnini,et al.  A Keyphrase-Based Approach to Summarization : the LAKE System at DUC-2005 , 2005 .

[12]  Paul Over,et al.  The Effects of Human Variation in DUC Summarization Evaluation , 2004 .

[13]  Douglas W. Oard,et al.  Extrinsic Evaluation of Automatic Metrics for Summarization , 2004 .

[14]  Atefeh Farzindar,et al.  CATS a topic-oriented multi-document summarization system at DUC 2005 , 2005 .

[15]  Daniel Marcu,et al.  An Evaluation Road Map for Summarization Research , 2000 .

[16]  Frank Schilder,et al.  TLR at DUC : Tree similarity , 2005 .

[17]  David Evans,et al.  Tracking and summarizing news on a daily basis with Columbia's Newsblaster , 2002 .

[18]  Yoshio Nakao.  How Small a Distinction among Summaries can the Evaluation Method Identify? , 2001, NTCIR.

[19]  Richard M. Schwartz,et al.  BBN/UMD at DUC-2004: Topiary , 2004 .

[20]  Hoa Trang Dang,et al.  Overview of DUC 2005 , 2005 .

[21]  Ani Nenkova,et al.  Automation of Summary Evaluation by the Pyramid Method , 2005 .