Overview of the EvaLatin 2020 Evaluation Campaign

This paper describes the first edition of EvaLatin, a campaign totally devoted to the evaluation of NLP tools for Latin. The two shared tasks proposed in EvaLatin 2020, i. e. Lemmatization and Part-of-Speech tagging, are aimed at fostering research in the field of language technologies for Classical languages. The shared dataset consists of texts taken from the Perseus Digital Library, processed with UDPipe models and then manually corrected by Latin experts. The training set includes only prose texts by Classical authors. The test set, alongside with prose texts by the same authors represented in the training set, also includes data relative to poetry and to the Medieval period. This also allows us to propose the Cross-genre and Cross-time subtasks for each task, in order to evaluate the portability of NLP tools for Latin across different genres and time periods. The results obtained by the participants for each task and subtask are presented and discussed.

[1]  Celano Giuseppe,et al.  A Gradient Boosting-Seq2Seq System for Latin POS Tagging and Lemmatization , 2020, LT4HALA.

[2]  Jeffrey A. Rydberg-Cox,et al.  The Perseus Project: a Digital Library for the Humanities , 2000 .

[3]  L. R. Palmer,et al.  The Latin Language , 1954 .

[4]  Winston Wu,et al.  JHUBC’s Submission to LT4HALA EvaLatin 2020 , 2020, LT4HALA.

[5]  Geoff Bacon.  Data-driven Choices in Neural Part-of-Speech Tagging for Latin , 2020, LT4HALA.

[6]  David Bamman,et al.  A Case Study in Treebank Collaboration and Comparison: Accusativus cum Infinitivo and Subordination in Latin , 2008, Prague Bull. Math. Linguistics.

[7]  Milan Straka,et al.  UDPipe at EvaLatin 2020: Contextualized Embeddings and Treebank Embeddings , 2020, LT4HALA.

[8]  David Bamman,et al.  The Ancient Greek and Latin Dependency Treebanks , 2011, Language Technology for Cultural Heritage.

[9]  Roland Vollgraf,et al.  FLAIR: An Easy-to-Use Framework for State-of-the-Art NLP , 2019, NAACL.

[10]  Marco Carlo Passarotti,et al.  The Project of the Index Thomisticus Treebank , 2019, Digital Classical Philology.

[11]  Slav Petrov,et al.  A Universal Part-of-Speech Tagset , 2011, LREC.

[12]  Hinrich Schütze,et al.  Efficient Higher-Order CRFs for Morphological Tagging , 2013, EMNLP.

[13]  Greta Franzini,et al.  Verba Bestiae: How Latin Conquered Heavy Metal , 2020, Multilingual Metal Music: Sociocultural, Linguistic and Literary Perspectives on Heavy Metal Lyrics.

[14]  Milan Straka,et al.  Tokenizing, POS Tagging, Lemmatizing and Parsing UD 2.0 with UDPipe , 2017, CoNLL.

[15]  Alexander Mehler,et al.  Voting for POS tagging of Latin texts: Using the flair of FLAIR to better Ensemble Classifiers by Example of Latin , 2020, LT4HALA.

[16]  Daniel Kondratyuk,et al.  75 Languages, 1 Model: Parsing Universal Dependencies Universally , 2019, EMNLP.