Paris and Stanford at EPE 2017: Downstream Evaluation of Graph-based Dependency Representations

We describe the STANFORD-PARIS and PARIS-STANFORD submissions to the 2017 Extrinsic Parser Evaluation (EPE) Shared Task. The purpose of this shared task was to evaluate dependency graphs on three downstream tasks. Through our submissions, we evaluated the usability of several representations derived from English Universal Dependencies (UD), as well as the Stanford Dependencies (SD), Predicate Argument Structure (PAS), and DM representations. We further compared two parsing strategies: Directly parsing to graph-based dependency representations and a two-stage process of first parsing to surface syntax trees and then applying rule-based augmentations to obtain the final graphs. Overall, our systems performed very well and our submissions ranked first and third. In our analysis, we find that the two-stage parsing process leads to better downstream performance, and that enhanced UD, a graph-based representation, consistently outperforms basic UD, a strict surface syntax representation, suggesting an advantage of enriched representations for downstream tasks.

[1]  Beatrice Santorini,et al.  Building a Large Annotated Corpus of English: The Penn Treebank , 1993, CL.

[2]  Stefan Riezler,et al.  A Comparison of Evaluation Metrics for a Broad-Coverage Stochastic Parser , 2003 .

[3]  Jan Hajic,et al.  The Prague Dependency Treebank , 2003 .

[4]  Jun'ichi Tsujii,et al.  Deep Linguistic Analysis for the Accurate Identification of Predicate-Argument Relations , 2004, COLING.

[5]  Andy Way,et al.  Long-Distance Dependency Resolution in Automatically Acquired Wide-Coverage PCFG-Based LFG Approximations , 2004, ACL.

[6]  Claire Cardie,et al.  Annotating Expressions of Opinions and Emotions in Language , 2005, Lang. Resour. Evaluation.

[7]  Stephan Oepen,et al.  Discriminant-Based MRS Banking , 2006, LREC.

[8]  Yusuke Miyao,et al.  Challenges in Mapping of Syntactic Representations for Framework-Independent Parser Evaluation , 2007 .

[9]  Mark Steedman,et al.  CCGbank: A Corpus of CCG Derivations and Dependency Structures Extracted from the Penn Treebank , 2007, CL.

[10]  Richard Johansson,et al.  The CoNLL 2008 Shared Task on Joint Parsing of Syntactic and Semantic Dependencies , 2008, CoNLL.

[11]  Christopher D. Manning,et al.  The Stanford Typed Dependencies Representation , 2008, CF+CDPE@COLING.

[12]  Josef van Genabith,et al.  Dependency Parsing Resources for French: Converting Acquired Lexical Functional Grammar F-Structure Annotations and Parsing F-Structures Directly , 2009, NODALIDA.

[13]  Jun'ichi Tsujii,et al.  Evaluating contributions of natural language parsers to protein–protein interaction extraction , 2008, Bioinform..

[14]  Sampo Pyysalo,et al.  Overview of BioNLP’09 Shared Task on Event Extraction , 2009, BioNLP@HLT-NAACL.

[15]  Josef van Genabith,et al.  LFG without C-structures , 2010 .

[16]  Sampo Pyysalo,et al.  Evaluating Dependency Representations for Event Extraction , 2010, COLING.

[17]  Jari Björne,et al.  EXTRACTING CONTEXTUALIZED COMPLEX BIOLOGICAL EVENTS WITH RICH GRAPH‐BASED FEATURE SETS , 2011, Comput. Intell..

[18]  Erik Velldal,et al.  UiO 2: Sequence-labeling Negation Using Dependency Features , 2012, *SEMEVAL.

[19]  Éric Villemonte de la Clergerie,et al.  A linguistically-motivated 2-stage Tree to Graph Transformation , 2012, TAG.

[20]  Benoît Sagot,et al.  The Alpage Architecture at the SANCL 2012 Shared Task: Robust Pre-Processing and Lexical Bridging for User-Generated Content Parsing , 2012, HLT-NAACL 2012.

[21]  Roser Morante,et al.  *SEM 2012 Shared Task: Resolving the Scope and Focus of Negation , 2012, *SEMEVAL.

[22]  Richard Johansson,et al.  Relational Features in Fine-Grained Opinion Analysis , 2013, CL.

[23]  Sigrid Klerke,et al.  Down-stream effects of tree-to-dependency conversions , 2013, HLT-NAACL.

[24]  Mihai Surdeanu,et al.  The Stanford CoreNLP Natural Language Processing Toolkit , 2014, ACL.

[25]  Stephan Oepen,et al.  Broad-Coverage Semantic Dependency Parsing , 2014 .

[26]  Stephan Oepen,et al.  SemEval 2014 Task 8: Broad-Coverage Semantic Dependency Parsing , 2014, *SEMEVAL.

[27]  Éric Villemonte de la Clergerie,et al.  Because Syntax Does Matter: Improving Predicate-Argument Structures Parsing with Syntactic Features , 2015, NAACL.

[28]  Alexis Nasr,et al.  Deeper syntax for better semantic parsing , 2016, COLING.

[29]  Christopher D. Manning,et al.  Enhanced English Universal Dependencies: An Improved Representation for Natural Language Understanding Tasks , 2016, LREC.

[30]  Sampo Pyysalo,et al.  Universal Dependencies v1: A Multilingual Treebank Collection , 2016, LREC.

[31]  Éric Villemonte de la Clergerie,et al.  Accurate Deep Syntactic Parsing of Graphs: The Case of French , 2016, LREC.

[32]  Richard Johansson.  EPE 2017 : The Trento – Gothenburg Opinion Extraction System , 2017 .

[33]  Kevin Duh,et al.  DyNet: The Dynamic Neural Network Toolkit , 2017, ArXiv.

[34]  Timothy Dozat,et al.  Stanford’s Graph-based Neural Dependency Parser at the CoNLL 2017 Shared Task , 2017, CoNLL.

[35]  Benoît Sagot,et al.  The ParisNLP entry at the ConLL UD Shared Task 2017: A Tale of a #ParsingTragedy , 2017, CoNLL Shared Task.

[36]  Timothy Dozat,et al.  Deep Biaffine Attention for Neural Dependency Parsing , 2016, ICLR.

[37]  Nizar Habash,et al.  CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies , 2017, CoNLL.

[38]  Bruno Guillaume,et al.  Enhanced UD Dependencies with Neutralized Diathesis Alternation , 2017, DepLing.

[39]  Jari Björne,et al.  The 2017 Shared Task on Extrinsic Parser Evaluation Towards a Reusable Community Infrastructure , .