The 2017 Shared Task on Extrinsic Parser Evaluation Towards a Reusable Community Infrastructure

The 2017 Shared Task on Extrinsic Parser Evaluation (EPE 2017) seeks to provide better estimates of the relative utility of different types of dependency representations for a variety of downstream applications that depend centrally on the analysis of grammatical structure. EPE 2017 defines a generalized notion of lexicalized syntactico-semantic dependency representations and provides a common interchange format to three state-of-the-art downstream applications, viz. biomedical event extraction, negation resolution, and fine-grained opinion analysis. As a first step towards building a generic and extensible infrastructure for extrinsic parser evaluation, the downstream applications have been generalized to support a broad range of diverese dependency representations (including divergent sentence and token boundaries) and to allow fully automated re-training and evaluation for a specific collection of parser outputs. Nine teams participated in EPE 2017, submitting 49 distinct runs that encompass many different families of dependency representations, distinct approaches to preprocessing and parsing, and various types and volumes of training data.