Data, Annotations and Measures in EASY the Evaluation Campaign for Parsers of French.

This paper presents the protocol of EASY the evaluation campaign for syntactic parsers of French in the EVALDA project of the TECHNOLANGUE program. We describe the participants, the corpus and its genre partitioning, the annotation scheme, which allows for the annotation of both constituents and relations, the evaluation methodology and, as an illustration, the results obtained by one participant on half of the corpus.

[1]  Dekang Lin,et al.  Dependency-Based Evaluation of Minipar , 2003 .

[2]  P. Blache,et al.  Une grille d'évaluation pour les analyseurs syntaxiques , 2003 .

[3]  Christian R. Huyck,et al.  Modifying Existing Annotated Corpora for General Comparative Evaluation of Parsing , 2007 .

[4]  Ted Briscoe,et al.  Relational evaluation schemes , 2002 .

[5]  Ted Briscoe,et al.  Parser evaluation: using a grammatical relation annotation scheme , 2003 .

[6]  Daniel Gildea,et al.  The Proposition Bank: An Annotated Corpus of Semantic Roles , 2005, CL.

[7]  Hans Uszkoreit,et al.  Proceedings of the Workshop `Beyond PARSEVAL --- Towards improved evaluation measures for parsing systems' at the 3rd International Conference on Language Resources and Evaluation , 2002 .

[8]  Beth Ann Hockey,et al.  An approach to Robust Partial Parsing and Evaluation Metrics , 1996 .

[9]  Alexandra Kinyon,et al.  Building a Treebank for French , 2000, LREC.

[10]  Lorna Balkan,et al.  Test Suites for Natural Language Processing , 1995, TC.

[11]  Sabine Brants,et al.  The TIGER Treebank , 2001 .

[12]  Rashmi Prasad,et al.  Annotating Discourse Connectives and Their Arguments , 2004, FCP@NAACL-HLT.

[13]  Ralph Grishman,et al.  A Procedure for Quantitatively Comparing the Syntactic Coverage of English Grammars , 1991, HLT.

[14]  Patrick Paroubek,et al.  PEAS, the first instantiation of a comparative framework for evaluating parsers of French , 2003, EACL.

[15]  Patrick Paroubek,et al.  The Ongoing Evaluation Campaign of Syntactic Parsing of French: EASY , 2004, LREC.

[16]  Beatrice Santorini,et al.  Building a Large Annotated Corpus of English: The Penn Treebank , 1993, CL.

[17]  Khalil Sima'an,et al.  Towards comparing parsers from different linguistic frameworks: An information theoretic approach , 2002 .

[18]  Ted Briscoe,et al.  Parser evaluation: a survey and a new proposal , 1998, LREC.