AutoSummENG and MeMoG in Evaluating Guided Summaries

Within this article, we present the application of the AutoSummENG and MeMoG methods within the TAC 2011 AESOP challenge. Both evaluation methods are based on n-gram graphs. The experiments indicate that both methods offer very high performance in different aspects of evaluation, without the need of deep analysis or preprocessing. The results also imply some interesting open problems and point to further directions of study, related to negative examples of good summaries.

[1]  Hoa Trang Dang,et al.  Overview of the TAC 2008 Update Summarization Task , 2008, TAC.

[2]  Dragos Stefan Munteanu,et al.  ParaEval: Using Paraphrases to Evaluate Summaries Automatically , 2006, NAACL.

[3]  Gonzalo Navarro,et al.  A guided tour to approximate string matching , 2001, CSUR.

[4]  George Giannakopoulos,et al.  TAC2011 MultiLing Pilot Overview , 2011, TAC.

[5]  Horst Bunke.  Error-Tolerant Graph Matching: A Formal Framework and Algorithms , 1998, SSPR/SPR.

[6]  George A. Vouros,et al.  Summarization system evaluation revisited: N-gram graphs , 2008, TSLP.

[7]  S. Szpakowicz,et al.  Vocabulary Usage in Newswire Summaries , 2004, Workshop On Text Summarization Branches Out.

[8]  Michele Banko,et al.  Using N-Grams To Understand the Nature of Summaries , 2004, HLT-NAACL.

[9]  Kathleen R. McKeown,et al.  Applying the Pyramid Method in the 2006 Document Understanding Conference , 2006 .

[10]  Γεώργιος Γιαννακόπουλος,et al.  Automatic Summarization from Multiple Documents , 2009 .

[11]  Hoa Trang Dang,et al.  Overview of DUC 2005 , 2005 .

[12]  Peter Willett,et al.  RASCAL: Calculation of Graph Similarity using Maximum Common Edge Subgraphs , 2002, Comput. J..

[13]  Eduard Hovy,et al.  Evaluating DUC 2005 using Basic Elements , 2005 .

[14]  Frank Schilder,et al.  A Metric for Automatically Evaluating Coherent Summaries via Context Chains , 2009, 2009 IEEE International Conference on Semantic Computing.

[15]  Eduard H. Hovy,et al.  Automatic Evaluation of Summaries Using N-gram Co-occurrence Statistics , 2003, NAACL.

[16]  Chin-Yew Lin,et al.  ROUGE: A Package for Automatic Evaluation of Summaries , 2004, ACL 2004.

[17]  George Giannakopoulos,et al.  Summarization System Evaluation Variations Based on N-Gram Graphs , 2010, TAC.