Parser evaluation: using a grammatical relation annotation scheme

We describe a recently developed corpus annotation scheme for evaluating parsers that avoids some of the shortcomings of current methods. The scheme encodes grammatical relations between heads and dependents, and has been used to mark up a new public-domain corpus of naturally occurring English text. We show how the corpus can be used to evaluate the accuracy of a robust parser, and relate the corpus to extant resources.

[1]  David M. Magerman.  Statistical Decision-Tree Models for Parsing , 1995, ACL.

[2]  Seth Kulick,et al.  Heuristics and Parse Ranking , 1995, IWPT.

[3]  Bob Carpenter,et al.  Probabilistic Parsing using Left Corner Language Models , 1997, IWPT.

[4]  R. Lee Humphreys,et al.  The linguistics of punctuation , 2004, Machine Translation.

[5]  Dekang Lin,et al.  Dependency-Based Evaluation of Minipar , 2003 .

[6]  Geoffrey Leech,et al.  Running a grammar factory: The production of syntactically analysed corpora or “treebanks” , 1991 .

[7]  Ronald M. Kaplan,et al.  Lexical Functional Grammar A Formal System for Grammatical Representation , 2004 .

[8]  Ted Briscoe,et al.  Developing and Evaluating a Probabilistic LR Parser of Part-of-Speech and Punctuation Labels , 1995, IWPT.

[9]  Satoshi Sekine,et al.  The Domain Dependence of Parsing , 1997, ANLP.

[10]  Eugene Charniak,et al.  Tree-Bank Grammars , 1996, AAAI/IAAI, Vol. 2.

[11]  Robert J. Gaizauskas,et al.  Evaluation in language and speech technology , 1998, Comput. Speech Lang..

[12]  David Elworthy,et al.  Does Baum-Welch Re-estimation Help Taggers? , 1994, ANLP.

[13]  John A. Carroll,et al.  Robust, applied morphological generation , 2000, INLG.

[14]  Dekang Lin,et al.  A dependency-based method for evaluating broad-coverage parsers , 1995, Natural Language Engineering.

[15]  Beth Ann Hockey,et al.  An approach to Robust Partial Parsing and Evaluation Metrics , 1996 .

[16]  Ann Bies,et al.  Bracketing Guidelines For Treebank II Style Penn Treebank Project , 1995 .

[17]  Lorna Balkan,et al.  TSNLP - Test Suites for Natural Language Processing , 1996, COLING.

[18]  Michael Collins,et al.  A New Statistical Parser Based on Bigram Lexical Dependencies , 1996, ACL.

[19]  Geoffrey Sampson,et al.  A proposal for improving the measurement of parse accuracy , 2000 .

[20]  Daniel Jurafsky,et al.  How Verb Subcategorization Frequencies Are Affected By Corpus Choice , 1998, COLING.

[21]  Ivan A. Sag,et al.  Book Reviews: Head-driven Phrase Structure Grammar and German in Head-driven Phrase-structure Grammar , 1996, CL.

[22]  Ted Briscoe,et al.  Can Subcategorisation Probabilities Help a Statistical Parser , 1998, VLC@COLING/ACL.

[23]  Beatrice Santorini,et al.  Building a Large Annotated Corpus of English: The Penn Treebank , 1993, CL.

[24]  Ted Briscoe,et al.  Parser evaluation: a survey and a new proposal , 1998, LREC.

[25]  Ralph Grishman,et al.  Evaluating Parsing Strategies Using Standardized Parse Files , 1992, ANLP.

[26]  Eric Atwell.  Comparative evaluation of grammatical annotation models , 1996 .

[27]  Ralph Grishman,et al.  Evaluating syntax performance of parser/grammars , 1991 .

[28]  Frederick B. Thompson,et al.  English for the computer , 1899, AFIPS '66 (Fall).