Les résultats de la campagne EASY d'évaluation des analyseurs syntaxiques du français

In this paper, we present the results of the EASY evaluation campaign on parsers of French. EASY has been the very first black-box comparative evaluation campaign for parsers of French, with objective quantitative performance measures. EASY was part of the TECHNO- LANGUE program of the Delegate Ministry of Research, jointly supported by the Delegate Mi- nistry of Industry and the ministry of Culture and Communication. After setting EASY in the context of parsing evaluation and giving an account of the campaign, we present the results ob- tained by 15 parsers according to syntactic relation and subcorpus genre. Then we propose some lessons to draw from this campaign, in particular about the evaluation protocole, the segmenting into linguistic units, the formalism and the annotation activities, the quality criteria to apply for data, annotations and results and finally about the notion of reference for parsing. We conclude by showing how EASY results extend through the PASSAGE project (ANR-06-MDCA-013), which has just started and whose aim is the automatic annotation of a large corpus by several parsers, the combination of which being parametrized by results stemming from evaluation. Mots-cles : analyseur syntaxique, evaluation, francais.

[1]  Mary P. Harper,et al.  SParseval: Evaluation Metrics for Parsing Speech , 2006, LREC.

[2]  Patrick Paroubek,et al.  The Ongoing Evaluation Campaign of Syntactic Parsing of French: EASY , 2004, LREC.

[3]  Patrick Paroubek,et al.  Data, Annotations and Measures in EASY the Evaluation Campaign for Parsers of French. , 2006, LREC.

[4]  Patrick Paroubek,et al.  PEAS, the first instantiation of a comparative framework for evaluating parsers of French , 2003, EACL.

[5]  P. Blache,et al.  Une grille d'évaluation pour les analyseurs syntaxiques , 2003 .

[6]  Christian R. Huyck,et al.  Modifying Existing Annotated Corpora for General Comparative Evaluation of Parsing , 2007 .

[7]  Beth Ann Hockey,et al.  An approach to Robust Partial Parsing and Evaluation Metrics , 1996 .

[8]  Ralph Grishman,et al.  A Procedure for Quantitatively Comparing the Syntactic Coverage of English Grammars , 1991, HLT.

[9]  Joseph Mariani,et al.  L"action GRACE d"evaluation de l"assignation des parties du discours pour le francais , 1999 .

[10]  Alexandra Kinyon,et al.  Building a Treebank for French , 2000, LREC.

[11]  Lorna Balkan,et al.  Test Suites for Natural Language Processing , 1995, TC.

[12]  Sabine Brants,et al.  The TIGER Treebank , 2001 .

[13]  Hans Uszkoreit,et al.  Proceedings of the Workshop `Beyond PARSEVAL --- Towards improved evaluation measures for parsing systems' at the 3rd International Conference on Language Resources and Evaluation , 2002 .

[14]  Beatrice Santorini,et al.  Building a Large Annotated Corpus of English: The Penn Treebank , 1993, CL.

[15]  Khalil Sima'an,et al.  Towards comparing parsers from different linguistic frameworks: An information theoretic approach , 2002 .