Using Mechanical Turk to Create a Corpus of Arabic Summaries

This paper describes the creation of a human-generated corpus of extractive Arabic summaries of a selection of Wikipedia and Arabic newspaper articles using Mechanical Turk?an online workforce. The purpose of this exercise was two-fold. First, it addresses a shortage of relevant data for Arabic natural language processing. Second, it demonstrates the application of Mechanical Turk to the problem of creating natural language resources. The paper also reports on a number of evaluations we have performed to compare the collected summaries against results obtained from a variety of automatic summarisation systems.

[1]  Hans Peter Luhn,et al.  The Automatic Creation of Literature Abstracts , 1958, IBM J. Res. Dev..

[2]  Phyllis B. Baxendale,et al.  Machine-Made Index for Technical Literature - An Experiment , 1958, IBM J. Res. Dev..

[3]  S. T. Dumais,et al.  Using latent semantic analysis to improve access to textual information , 1988, CHI '88.

[4]  Xin Liu,et al.  Generic text summarization using relevance measure and latent semantic analysis , 2001, SIGIR '01.

[5]  Hinrich Schütze,et al.  Book Reviews: Foundations of Statistical Natural Language Processing , 1999, CL.

[6]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[7]  J. Steinberger,et al.  Using Latent Semantic Analysis in Text Summarization and Summary Evaluation , 2004 .

[8]  Chin-Yew Lin,et al.  ROUGE: A Package for Automatic Evaluation of Summaries , 2004, ACL 2004.

[9]  Richard M. Schwartz,et al.  Task-based evaluation of text summarization using Relevance Prediction , 2007, Inf. Process. Manag..

[10]  Kadri Hacioglu,et al.  Automatic Processing of Modern Standard Arabic Text , 2007 .

[11]  Brendan T. O'Connor,et al.  Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks , 2008, EMNLP.

[12]  Chafic Mokbel,et al.  MEDAR: Collaboration between European and Mediterranean Arabic Partners to Support the Development of Language Technology for Arabic , 2008, LREC.

[13]  Wei-Pang Yang,et al.  iSpreadRank: Ranking sentences for extraction-based summarization using feature weight propagation in the sentence similarity network , 2008, Expert Syst. Appl..

[14]  Bassam H. Hammo,et al.  Evaluation of Query-Based Arabic Text Summarization System , 2008, 2008 International Conference on Natural Language Processing and Knowledge Engineering.

[15]  Saroj Kaushik,et al.  Automatic Text Summarization , 2008 .

[16]  George A. Vouros,et al.  Summarization system evaluation revisited: N-gram graphs , 2008, TSLP.

[17]  Feifan Liu,et al.  Correlation between ROUGE and Human Evaluation of Extractive Meeting Summaries , 2008, ACL.

[18]  Chris Callison-Burch,et al.  Fast, Cheap, and Creative: Evaluating Translation Quality Using Amazon’s Mechanical Turk , 2009, EMNLP.

[19]  Chafic Mokbel,et al.  Arabic Language Resources and Tools for Speech and Natural Language , 2009 .

[20]  Vasudeva Varma,et al.  Sentence Position revisited: A robust light-weight Update Summarization ‘baseline’ Algorithm , 2009 .

[21]  Udo Kruschwitz,et al.  Experimenting with Automatic Text Summarisation for Arabic , 2009, LTC.

[22]  Halil Kilicoglu,et al.  Automatic summarization of MEDLINE citations for evidence-based medical treatment: A topic-oriented evaluation , 2009, J. Biomed. Informatics.

[23]  Simon M. Lucas,et al.  Sentence-Level Attachment Prediction , 2010, IRFC.