Capitalizing on a TREC Track to Build a Tweet Summarization Dataset

Today there is a lack of standard collection for automatic tweet summarization evaluation. The construction of such a large dataset is very tedious. In this paper, we check whether the dataset proposed for the TREC Incident Streams track, which was not created for automatic summary generation, could be used in this way. Indeed, when filtering the TREC Incident Streams (IS) dataset with the assessors’ annotations, it appears to respect the citeria identified in the literature related to automatic summarization. For this, we studied the TREC IS dataset and then proposed a subset summarizing each event, based on the assessors’ annotations. This subset is evaluated according to the criteria previously mentioned. Several widely used state-of-the-art models for automatic text summarization, adapted to tweet summarization, were finally tested on the proposed dataset. The code, the annotations and the results are provided on our Github.

[1]  Alon Lavie,et al.  Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems , 2011, WMT@EMNLP.

[2]  Cody Buntain,et al.  TREC Incident Streams: Finding Actionable Information on Social Media , 2019, ISCRAM.

[3]  R. P. Fishburne,et al.  Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel , 1975 .

[4]  J. Chall,et al.  A FORMULA FOR PREDICTING READABILITY , 1948 .

[5]  E A Smith,et al.  Automated readability index. , 1967, AMRL-TR. Aerospace Medical Research Laboratories.

[6]  Dragomir R. Radev,et al.  Centroid-based summarization of multiple documents , 2004, Inf. Process. Manag..

[7]  Rim Faiz,et al.  Conversational based method for tweet contextualization , 2017, Vietnam Journal of Computer Science.

[8]  Xiaojun Wan,et al.  Multi-document summarization using cluster-based link analysis , 2008, SIGIR '08.

[9]  Jun-Ping Ng,et al.  Better Summarization Evaluation with Word Embeddings for ROUGE , 2015, EMNLP.

[10]  Gilles Hubert,et al.  IRIT at TREC 2019: Incident Streams and Complex Answer Retrieval Tracks , 2019, TREC.

[11]  Hui Lin,et al.  Multi-document Summarization via Budgeted Maximization of Submodular Functions , 2010, NAACL.

[12]  Dragomir R. Radev,et al.  Introduction to the Special Issue on Summarization , 2002, CL.

[13]  Fernando Diaz,et al.  Extracting information nuggets from disaster- Related messages in social media , 2013, ISCRAM.

[14]  Jaime Carbonell,et al.  Multi-Document Summarization By Sentence Extraction , 2000 .

[15]  M. Coleman,et al.  A computer readability formula designed for machine scoring. , 1975 .

[16]  Krys J. Kochut,et al.  Text Summarization Techniques: A Brief Survey , 2017, International Journal of Advanced Computer Science and Applications.

[17]  Lei Huang,et al.  Modeling Document Summarization as Multi-objective Optimization , 2010, 2010 Third International Symposium on Intelligent Information Technology and Security Informatics.

[18]  Vishal Gupta,et al.  Recent automatic text summarization techniques: a survey , 2016, Artificial Intelligence Review.

[19]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[20]  Rada Mihalcea,et al.  TextRank: Bringing Order into Text , 2004, EMNLP.

[21]  Craig MacDonald,et al.  Overview of the TREC-2012 Microblog Track , 2012, Text Retrieval Conference.

[22]  Tao Li,et al.  Multi-document summarization via submodularity , 2012, Applied Intelligence.

[23]  Carlos Castillo,et al.  What to Expect When the Unexpected Happens: Social Media Communications Across Crises , 2015, CSCW.

[24]  Ani Nenkova,et al.  The Pyramid Method: Incorporating human content selection variation in summarization evaluation , 2007, TSLP.

[25]  Jimmy J. Lin,et al.  Overview of the TREC 2017 Real-Time Summarization Track , 2017, TREC.

[26]  Muhammad Imran,et al.  Identifying Sub-events and Summarizing Disaster-Related Information from Microblogs , 2018, SIGIR.

[27]  Patrice Bellot,et al.  INEX Tweet Contextualization task: Evaluation, results and lesson learned , 2016, Inf. Process. Manag..

[28]  R. Gunning.  The Technique of Clear Writing. , 1968 .

[29]  Dragomir R. Radev,et al.  LexPageRank: Prestige in Multi-Document Text Summarization , 2004, EMNLP.

[30]  Minh-Tien Nguyen,et al.  TSix: A Human-involved-creation Dataset for Tweet Summarization , 2018, LREC.

[31]  Niloy Ganguly,et al.  Summarizing Situational Tweets in Crisis Scenarios: An Extractive-Abstractive Approach , 2019, IEEE Transactions on Computational Social Systems.

[32]  Muhammad Imran,et al.  Twitter as a Lifeline: Human-annotated Twitter Corpora for NLP of Crisis-related Messages , 2016, LREC.

[33]  G. Spache,et al.  A New Readability Formula for Primary-Grade Reading Materials , 1953, The Elementary School Journal.

[34]  Josiane Mothe,et al.  A survey on evaluation of summarization methods , 2019, Inf. Process. Manag..

[35]  Jugal K. Kalita,et al.  Summarization of Twitter Microblogs , 2014, Comput. J..

[36]  Jianwu Dang,et al.  Twitter summarization with social-temporal context , 2016, World Wide Web.

[37]  Chin-Yew Lin,et al.  ROUGE: A Package for Automatic Evaluation of Summaries , 2004, ACL 2004.

[38]  Jimmy J. Lin,et al.  Overview of the TREC-2013 Microblog Track , 2013, TREC.