On the subjectivity of human-authored summaries

Human-generated summaries are a blend of content and style, bound by the task restrictions, but are ‘subject to subjectiveness’ of the individuals summarising the documents. We study the impact of various facets that cause subjectivity such as brevity, information content and information coverage on human-authored summaries. The scale of subjectivity is quantitatively measured among various summaries using a question–answer-based cross-comprehension test. The test evaluates summaries for meaning rather than exact words based on questions, framed by the summary authors, derived from the summary. The number of questions that cannot be answered after reading the candidate summary reflects its subjectivity. The qualitative analysis of the outcome of the cross-comprehension test shows the relationship between the length of a summary, information content and nature of questions framed by the summary author.

[1]  Eduard Hovy,et al.  The Potential and Limitations of Automatic Sentence Extraction for Summarization , 2003, HLT-NAACL 2003.

[2]  Dragomir R. Radev,et al.  Centroid-based summarization of multiple documents , 2004, Inf. Process. Manag..

[3]  Michele Banko,et al.  Event-Centric Summary Generation , 2004 .

[4]  Jun-ichi Fukumoto,et al.  Automated Summarization Evaluation with Basic Elements. , 2006, LREC.

[5]  Paul Over,et al.  Intrinsic Evaluation of Generic News Text Summarization Systems , 2003 .

[6]  Mark Liberman,et al.  THE TDT-2 TEXT AND SPEECH CORPUS , 1999 .

[7]  Sadaoki Furui,et al.  Sentence extraction-based presentation summarization techniques and evaluation metrics , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[8]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[9]  Lynette Hirschman,et al.  EVALUATING CONTENT EXTRACTION FROM AUDIO SOURCES , 1999 .

[10]  Jimmy J. Lin,et al.  Different Structures for Evaluating Answers to Complex Questions: Pyramids Won’t Topple, and Neither Will Human Assessors , 2007, ACL.

[11]  Walter Kintsch,et al.  Toward a model of text comprehension and production. , 1978 .

[12]  Edward T. Cremmins.  The Art of Abstracting. , 1982 .

[13]  McKeownKathleen,et al.  The Pyramid Method , 2007 .

[14]  Sadaoki Furui,et al.  Evaluation method for automatic speech summarization , 2003, INTERSPEECH.

[15]  Jimmy J. Lin,et al.  Will Pyramids Built of Nuggets Topple Over? , 2006, NAACL.

[16]  Yoshihiko Gotoh,et al.  Relative evaluation of informativeness in machine generated summaries , 2007, INTERSPEECH.

[17]  Jennifer Rowley,et al.  Abstracting and indexing , 1982 .

[18]  Mark T. Maybury,et al.  Automatic Summarization , 2002, Computational Linguistics.

[19]  Eduard H. Hovy,et al.  Automatic Evaluation of Summaries Using N-gram Co-occurrence Statistics , 2003, NAACL.

[20]  Simone Teufel,et al.  Examining the consensus between human summaries: initial experiments with factoid analysis , 2003, HLT-NAACL 2003.

[21]  Heidi Christensen,et al.  Multi-stage compaction approach to broadcast news summarisation , 2005, INTERSPEECH.

[22]  María Pinto Molina.  Documentary abstracting: toward a methodological model , 1995 .

[23]  Ani Nenkova,et al.  Automatic Summarization , 2011, ACL.

[24]  Hans van Halteren,et al.  Evaluating Information Content by Factoid Analysis: Human annotation and stability , 2004, EMNLP.

[25]  Elisabeth Neugebauer,et al.  Professional summarizing: no cognitive simulation without observation , 1998 .

[26]  Hongyan Jing,et al.  Using Hidden Markov Modeling to Decompose Human-Written Summaries , 2002, Computational Linguistics.

[27]  Ani Nenkova,et al.  The Pyramid Method: Incorporating human content selection variation in summarization evaluation , 2007, TSLP.

[28]  Ani Nenkova,et al.  Evaluating Content Selection in Summarization: The Pyramid Method , 2004, NAACL.