Evaluation of Out-of-Domain Dependency Parsing for its Application in a Digital Humanities Project

In this paper we evaluate the out-ofdomain performance of six commonly used parsers. Our work is situated in a digital humanities project, in which we are interested in the analysis of various text types such as literature or academic text, for which we do not have sufficient training data available. In our evaluation, we focus on those dependency labels that are most relevant to further analysis for research questions in the humanities. The results show good overall performance with Mate being the most successful parser. In general, however, the performance on some labels of interest to humanist scholars (e. g. non-verbal predicates) is rather poor.

[1]  Christian Chiarcos,et al.  A New Hybrid Dependency Parser for German , 2009 .

[2]  Gerold Schneider,et al.  Exploiting Synergies Between Open Resources for German Dependency Parsing, POS-tagging, and Morphological Analysis , 2013, RANLP.

[3]  Slav Petrov,et al.  Overview of the 2012 Shared Task on Parsing the Web , 2012 .

[4]  Bernd Bohnet,et al.  Very high accuracy and fast dependency parsing is not a contradiction , 2010, COLING 2010.

[5]  Mihai Surdeanu,et al.  Ensemble Models for Dependency Parsing: Cheap and Good? , 2010, HLT-NAACL.

[6]  Wolfgang Menzel,et al.  Because Size Does Matter: The Hamburg Dependency Treebank , 2014, LREC.

[7]  Jun'ichi Tsujii,et al.  Dependency Parsing and Domain Adaptation with LR Models and Parser Ensembles , 2007, EMNLP.

[8]  Ruken Cakici,et al.  Multi-lingual Dependency Parsing with Incremental Integer Linear Programming , 2006, CoNLL.

[9]  Joakim Nivre,et al.  An Efficient Algorithm for Projective Dependency Parsing , 2003, IWPT.

[10]  Daniel Gildea,et al.  Corpus Variation and Parser Performance , 2001, EMNLP.

[11]  Yannick Versley.  Parser evaluation across Text Types , 2005 .

[12]  Niels Ott,et al.  Evaluating Dependency Parsing Performance on German Learner Language , 2010 .

[13]  J. Fleiss.  Measuring nominal scale agreement among many raters. , 1971 .

[14]  Regina Barzilay,et al.  Low-Rank Tensors for Scoring Dependency Structures , 2014, ACL.

[15]  Josef van Genabith,et al.  #hardtoparse: POS Tagging and Parsing the Twitterverse , 2011, Analyzing Microtext.

[16]  Fernando Pereira,et al.  Multilingual Dependency Analysis with a Two-Stage Discriminative Parser , 2006, CoNLL.

[17]  Xavier Carreras,et al.  Experiments with a Higher-Order Projective Dependency Parser , 2007, EMNLP.

[18]  Noah A. Smith,et al.  Turning on the Turbo: Fast Third-Order Non-Projective Turbo Parsers , 2013, ACL.

[19]  Wolfgang Menzel,et al.  Predictive Incremental Parsing and its Evaluation , 2011, DepLing.

[20]  Kilian A. Foth.  Eine umfassende Constraint-Dependenz-Grammatik des Deutschen , 2006 .

[21]  Heike Zinsmeister,et al.  Exploring Ensemble Dependency Parsing to Reduce Manual Annotation Workload , 2017, GSCL.

[22]  Christian Chiarcos,et al.  Von der Form zur Bedeutung: Texte automatisch verarbeiten/From Form to Meaning: Processing Texts Automatically , 2009 .

[23]  Sabine Buchholz,et al.  CoNLL-X Shared Task on Multilingual Dependency Parsing , 2006, CoNLL.

[24]  Wolfgang Lezius,et al.  TIGER: Linguistic Interpretation of a German Corpus , 2004 .