Centering Theory for Evaluation of Coherence in Computer-Aided Summaries

This paper investigates a new evaluation method for assessing the coherence of computer-aided summaries, justified by the inappropriacy of existing evaluation methods for this task. It develops a metric for Centering Theory (CT), a theory of local coherence and salience, to measure coherence in pairs of extracts and abstracts produced in a computer-aided summarisation environment. 100 news text summaries (50 pairs of extracts and their corresponding abstracts) are analysed using CT and the metric is applied to obtain a score for each summary; the summary with the higher score out of a pair is considered more coherent. Human judgement is also obtained to allow a comparison with the CT evaluation to assess the validity of the development of CT as a useful evaluation metric in computer-aided summarisation.

[1]  M. Ferguson,et al.  Automatic Evaluation , 2009 .

[2]  Megumi Kameyama,et al.  Intrasentential Centering: A Case Study , 1997, ArXiv.

[3]  Robert L. Donaway,et al.  A Comparison of Rankings Produced by Summarization Evaluation Measures , 2000 .

[4]  Laura Hasler,et al.  FROM EXTRACTS TO ABSTRACTS: HUMAN SUMMARY PRODUCTION OPERATIONS FOR COMPUTER-AIDED SUMMARISATION , 2007 .

[5]  J. Kagan.  The concept of identification. , 1958, Psychological review.

[6]  Udo Hahn,et al.  Functional Centering - Grounding Referential Coherence in Information Structure , 1999, Comput. Linguistics.

[7]  Constantin Orasan,et al.  CAST: A computer-aided summarisation tool , 2003, EACL.

[8]  Chin-Yew Lin,et al.  ROUGE: A Package for Automatic Evaluation of Summaries , 2004, ACL 2004.

[9]  Ruslan Mitkov,et al.  The Oxford handbook of computational linguistics , 2003 .

[10]  Constantin Orasan,et al.  Computer-aided summarisation – what the user really wants , 2006, LREC.

[11]  Scott Weinstein,et al.  Centering: A Framework for Modeling the Local Coherence of Discourse , 1995, CL.

[12]  Jean-Luc Minel,et al.  How to Appreciate the Quality of Automatic Text Summarization? Examples of FAN and MLUCE Protocols and their Results on SERAPHIN , 1997, ACL 1997.

[13]  R. Gunning.  The Technique of Clear Writing. , 1968 .

[14]  Mark T. Maybury,et al.  Automatic Summarization , 2002, Computational Linguistics.

[15]  M. Walker,et al.  Centering in Naturally-Occurring Discourse: An Overview , 2007 .

[16]  R. P. Fishburne,et al.  Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel , 1975 .

[17]  Ani Nenkova,et al.  Automation of Summary Evaluation by the Pyramid Method , 2005 .

[18]  Udo Hahn,et al.  Functional Centering , 1996, ACL.

[19]  Rodger Kibble.  A Reformulation of Rule 2 of Centering Theory , 2001, Computational Linguistics.

[20]  Constantin Orasan,et al.  Building better corpora for summarisation , 2003 .

[21]  Chris Mellish,et al.  Evaluating Centering-Based Metrics of Coherence , 2004, ACL.

[22]  Horacio Saggion,et al.  Concept Identification and Presentation in the Context of Technical Text Summarization , 2000 .

[23]  Carl Pollard,et al.  A Centering Approach to Pronouns , 1987, ACL.

[24]  Barbara Di Eugenio,et al.  Centering: A Parametric Theory and Its Instantiations , 2004, Computational Linguistics.

[25]  Mirella Lapata,et al.  Automatic Evaluation of Text Coherence: Models and Representations , 2005, IJCAI.

[26]  L. Hasler.  An investigation into the use of Centering transitions for summarisation , 2003 .

[27]  Ani Nenkova,et al.  Evaluating Content Selection in Summarization: The Pyramid Method , 2004, NAACL.

[28]  Nikiforos Karamanis,et al.  Entity coherence for descriptive text structuring , 2004 .