Corpus annotation with paraphrase types: new annotation scheme and inter-annotator agreement measures

Paraphrase corpora annotated with the types of paraphrases they contain constitute an essential resource for the understanding of the phenomenon of paraphrasing and the improvement of paraphrase-related systems in natural language processing. In this article, a new annotation scheme for paraphrase-type annotation is set out, together with newly created measures for the computation of inter-annotator agreement. Three corpora different in nature and in two languages have been annotated using this infrastructure. The annotation results and the inter-annotator agreement scores for these corpora are proof of the adequacy and robustness of our proposal.

[1]  Horacio Rodríguez,et al.  Relational paraphrase acquisition from Wikipedia: The WRPA method and corpus† , 2013, Natural Language Engineering.

[2]  Jun-ichi Fukumoto,et al.  Automated Summarization Evaluation with Basic Elements. , 2006, LREC.

[3]  Eduard H. Hovy,et al.  Automatic Evaluation of Summaries Using N-gram Co-occurrence Statistics , 2003, NAACL.

[4]  Ani Nenkova,et al.  Evaluating Content Selection in Summarization: The Pyramid Method , 2004, NAACL.

[5]  Julio Gonzalo,et al.  MT Evaluation: Human-Like vs. Human Acceptable , 2006, ACL.

[6]  Nitin Madnani,et al.  Generating Phrasal and Sentential Paraphrases: A Survey of Data-Driven Methods , 2010, CL.

[7]  Benno Stein,et al.  An Evaluation Framework for Plagiarism Detection , 2010, COLING.

[8]  K. B. Hafner,et al.  On assessing interrater agreement for multiple attribute responses. , 1989, Biometrics.

[9]  Alberto Barrón-Cedeño,et al.  Plagiarism Meets Paraphrasing: Insights for the Next Generation in Automatic Plagiarism Detection , 2013, CL.

[10]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[11]  Regina Barzilay,et al.  Extracting Paraphrases from a Parallel Corpus , 2001, ACL.

[12]  Eneko Agirre,et al.  SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity , 2012, *SEMEVAL.

[13]  William B. Dolan,et al.  Collecting Highly Parallel Data for Paraphrase Evaluation , 2011, ACL.

[14]  Horacio Rodríguez,et al.  Is This a Paraphrase? What Kind? Paraphrase Boundaries and Typology , 2014 .

[15]  Marta Recasens,et al.  On Paraphrase and Coreference , 2010, Computational Linguistics.

[16]  Guillaume Wisniewski,et al.  Mining Naturally-occurring Corrections and Paraphrases from Wikipedia’s Revision History , 2022, LREC.

[17]  Mark Dras,et al.  Tree Edit Distance as a Baseline Approach for Paraphrase Representation , 2012, Proces. del Leng. Natural.

[18]  Ido Dagan,et al.  Investigating a Generic Paraphrase-Based Approach for Relation Extraction , 2006, EACL.

[19]  Chin-Yew Lin,et al.  ORANGE: a Method for Evaluating Automatic Evaluation Metrics for Machine Translation , 2004, COLING.

[20]  J. Fleiss.  Statistical methods for rates and proportions , 1974 .

[21]  Mirella Lapata,et al.  Constructing Corpora for the Development and Evaluation of Paraphrase Systems , 2008, CL.

[22]  Robert Dale,et al.  A Framework for Evaluating Text Correction , 2012, LREC.

[23]  Chris Brockett,et al.  Automatically Constructing a Corpus of Sentential Paraphrases , 2005, IJCNLP.

[24]  C. Fuchs.  Paraphrases prédicatives et contraintes énonciatives in Lexique et paraphrase. , 1988 .

[25]  Jasmina Milićević,et al.  La paraphrase : modélisation de la paraphrase langagière , 2007 .

[26]  Houda Bouamor,et al.  Local modifications and paraphrases in Wikipedia's revision history , 2011, Proces. del Leng. Natural.

[27]  Eduard Hovy,et al.  Learning paraphrases from text , 2009 .

[28]  Annie Zaenen.  Mark-up Barking Up the Wrong Tree , 2006, Computational Linguistics.

[29]  Horacio Rodríguez,et al.  CoCo, a web interface for corpora compilation , 2009 .

[30]  Hwee Tou Ng,et al.  PEM: A Paraphrase Evaluation Metric Exploiting Parallel Texts , 2010, EMNLP.

[31]  Adam Kilgarriff,et al.  Helping Our Own: The HOO 2011 Pilot Shared Task , 2011, ENLG.