Evaluation of Automatic Text Summarizations based on Human Summaries

Abstract The goal of this paper is to compare summaries generated by different automatic text summarization methods and those generated by human beings. To achieve this end, we did two series of experiments: in the first one, we employed automatically produced extractive summaries; in the second one, manually-produced summaries obtained by several English teachers were used. Our automatic summaries were obtained using Fuzzy method and Vector approach. Using Rouge evaluation system, we compared the manually-produced summaries and the automatically-produced ones. Rouge evaluation of generated summaries indicated the superiority of summaries produced by humans over the automatically produced summaries. On the other hand, the comparison between the generated summaries showed that summaries produced by Fuzzy method were much more acceptable and understandable compared to summaries produced by Vector approach. This can provide support for the replacement of manually generated summaries by summaries produced using Fuzzy method in certain cases where real time summaries are needed.

[1]  Esfandiar Eslami,et al.  Optimizing Text Summarization Based on Fuzzy Logic , 2008, Seventh IEEE/ACIS International Conference on Computer and Information Science (icis 2008).

[2]  J. Buckley,et al.  An Introduction to Fuzzy Logic and Fuzzy Sets , 2002 .

[3]  Alex Alves Freitas,et al.  Automatic Text Summarization Using a Machine Learning Approach , 2002, SBIA.

[4]  Brian Roark,et al.  Query-focused summarization by supervised sentence ranking and skewed word distributions , 2006 .

[5]  B.A. Kiani,et al.  Intelligent Extractive Text Summarization Using Fuzzy Inference Systems , 2006, 2006 IEEE International Conference on Engineering of Intelligent Systems.

[6]  Christopher C. Yang,et al.  Fractal summarization: summarization based on fractal theory , 2003, SIGIR '03.

[7]  Ani Nenkova,et al.  Automatic Summarization , 2011, ACL.

[8]  Mark T. Maybury,et al.  Advances in Automatic Text Summarization , 1999 .

[9]  Gerard Salton,et al.  Term-Weighting Approaches in Automatic Text Retrieval , 1988, Inf. Process. Manag..

[10]  Eduard H. Hovy,et al.  Automatic Evaluation of Summaries Using N-gram Co-occurrence Statistics , 2003, NAACL.

[11]  Kathleen R. McKeown,et al.  Generating natural language summaries from multiple on-line sources , 1998 .

[12]  Noémie Elhadad.  User-sensitive text summarization thesis summary , 2004, AAAI 2004.

[13]  Eduard Hovy,et al.  Manual and automatic evaluation of summaries , 2002, ACL 2002.

[14]  Louisa Ferrier.  A Maximum Entropy Approach to Text Summarization , 2001 .

[15]  Yuji Matsumoto,et al.  A new approach to unsupervised text summarization , 2001, SIGIR '01.

[16]  Rajkumar Roy,et al.  Advances in Soft Computing , 2018, Lecture Notes in Computer Science.

[17]  Xin Liu,et al.  Creating generic text summaries , 2001, Proceedings of Sixth International Conference on Document Analysis and Recognition.

[18]  Dragomir R. Radev,et al.  Generating Natural Language Summaries from Multiple On-Line Sources , 1998, CL.

[19]  K. Kaikhah.  Automatic text summarization with neural networks , 2004, 2004 2nd International IEEE Conference on 'Intelligent Systems'. Proceedings (IEEE Cat. No.04EX791).