A Protocol for Evaluating Analyzers of Syntax (PEAS)

Providing a comparative framework for parsers is a task that has already been tried in the past, e.g. (Abeille, 1991), (Atwell and Sutcliffe, 1997), (Black et al., 1991), and studied in the literature (Black, 1993), (Black, 1994), (Carroll et al., 1998), (Gaizauskas et al., 1998), (WEPS-98, ), (Mengel and Lezius, 2000), but mainly for English. In this paper, we present PEAS: a Protocol for Evaluating Analyzers of Syntax (in French: Protocole d’Evaluation pour les Analyseurs Syntaxiques), based on an ongoing experiment at LIMSI which aims at developing and testing a generic quantitative black-box evaluation protocol for parsers of French. Two fully operational parsers will be used to test the evaluation protocol; they are: the parser (Giguet and Vergne, 1997) developed at GREYC (Caen University) and the latest version of the parser developed at Rank Xerox Research Center in Grenoble (Ait-Mokhtar and Chanod, 1997)

[1]  Gaston Leroux,et al.  Le mystère de la chambre jaune , 1907 .

[2]  Ralph Grishman,et al.  A Procedure for Quantitatively Comparing the Syntactic Coverage of English Grammars , 1991, HLT.

[3]  Ralph Grishman,et al.  Evaluating syntax performance of parser/grammars , 1991 .

[4]  Eric Wehrli,et al.  The Ips System , 1992, COLING.

[5]  Beatrice Santorini,et al.  Building a Large Annotated Corpus of English: The Penn Treebank , 1993, CL.

[6]  Daniel Dominic Sleator,et al.  Parsing English with a Link Grammar , 1995, IWPT.

[7]  Ann Bies,et al.  The Penn Treebank: Annotating Predicate Argument Structure , 1994, HLT.

[8]  Dekang Lin,et al.  A dependency-based method for evaluating broad-coverage parsers , 1995, Natural Language Engineering.

[9]  Steven P. Abney Partial parsing via finite-state cascades , 1996, Natural Language Engineering.

[10]  Lorna Balkan,et al.  TSNLP - Test Suites for Natural Language Processing , 1996, COLING.

[11]  Ted Briscoe,et al.  SPARKLE Work Package 1: Specification of Phrasal Parsing. Final Report , 1997 .

[12]  Jean-Pierre Chanod,et al.  Incremental Finite-State Parsing , 1997, ANLP.

[13]  Ted Briscoe,et al.  Parser evaluation: a survey and a new proposal , 1998, LREC.

[14]  Jacques Vergne Syntactic Analysis of Unrestricted French , 1998 .

[15]  Christian R. Huyck,et al.  A scheme for comparative evaluation of diverse parsing systems , 1998 .

[16]  Klaus Netter,et al.  DiET in the context of MT evaluation , 1998 .

[17]  P. Paroubek,et al.  Human Language Technologies Evaluation in the European Framework , 1999 .

[18]  Vito Pirrelli,et al.  Where Opposites Meet. A Syntactic Meta-scheme for Corpus Annotation and Parsing Evaluation , 2000, LREC.

[19]  Alexandra Kinyon,et al.  Building a Treebank for French , 2000, LREC.

[20]  Wolfgang Lezius,et al.  An XML-based Representation Format for Syntactically Annotated Corpora , 2000, LREC.

[21]  Patrick Paroubek Language Resources as by-Product of Evaluation: The MULTITAG Example , 2000, LREC.

[22]  Nancy Ide,et al.  A Common Framework for Syntactic Annotation , 2001, ACL.

[23]  Anne Abeillé,et al.  Un corpus français arboré : quelques interrogations , 2001, JEPTALNRECITAL.