A Multilingual Parallel Parsed Corpus as Gold Standard for Grammatical Inference Evaluation

In this article we investigate how (computational) grammar inference systems are evaluated and how the evaluation procedure can be improved. First, we describe the currently used evaluation methods and look at the advantages and disadvantages of each method. The main problems of the methods are: the dependency on language experts, the influence of the annotation scheme of language data, and the language dependency of the evaluation. We then propose a new method that will allow for an evaluation independently of language and annotation scheme. This method requires (syntactically) structured corpora in multiple languages to test for language independency of the grammatical inference system and corpora structured using different annotation schemes to diminish the influence the annotation has on the evaluation.

[1]  M. A. R T A P A L,et al.  The Penn Chinese TreeBank: Phrase structure annotation of a large corpus , 2005, Natural Language Engineering.

[2]  Gertjan van Noord,et al.  The Alpino Dependency Treebank , 2001, CLIN.

[3]  Katsuhiko Nakamura,et al.  Incremental Learning of Context Free Grammars , 2002, ICGI.

[4]  Eric Atwell,et al.  Rationale for a multilingual corpus for machine translation evaluation , 2003 .

[5]  Jeffrey D. Ullman,et al.  Introduction to Automata Theory, Languages and Computation , 1979 .

[6]  Wojciech Skut,et al.  An Annotation Scheme for Free Word Order Languages , 1997, ANLP.

[7]  Menno van Zaanen,et al.  Bootstrapping structure into language : alignment-based learning , 2001, ArXiv.

[8]  Katsuhiko Nakamura,et al.  Synthesizing Context Free Grammars from Sample Strings Based on Inductive CYK Algorithm , 2000, ICGI.

[9]  Menno van Zaanen,et al.  Alignment-based learning versus emile: A comparison , 2001 .

[10]  Eric Atwell,et al.  A comparative evaluation of modern English corpus grammatical annotation schemes , 2000 .

[11]  Petya Osenova,et al.  The Bulgarian HPSG Treebank : Specialization of the Annotation Scheme , 2003 .

[12]  Dan Klein,et al.  A Generative Constituent-Context Model for Improved Grammar Induction , 2002, ACL.

[13]  Anthony McEnery,et al.  Proceedings of the corpus linguistics 2003 conference. , 2001 .

[14]  Azriel Rosenfeld,et al.  Grammatical inference by hill climbing , 1976, Inf. Sci..

[15]  Ralph Grishman,et al.  A Procedure for Quantitatively Comparing the Syntactic Coverage of English Grammars , 1991, HLT.

[16]  Andreas Stolcke,et al.  Inducing Probabilistic Grammars by Bayesian Model Merging , 1994, ICGI.

[17]  J. Wolff,et al.  Language Acquisition and the Discovery of Phrase Structure , 1980, Language and speech.

[18]  Alexander Clark.  Unsupervised induction of stochastic context-free grammars using distributional clustering , 2001, CoNLL.

[19]  J. Gerard Wolfp,et al.  Language Acquisition and the Discovery of Phrase Structure , 1980 .

[20]  Hervé Déjean.  ALLiS: a Symbolic Learning System for Natural Language Learning , 2000, CoNLL/LLL.

[21]  Andrew Roberts,et al.  The use of corpora for automatic evaluation of grammar inference systems , 2003 .

[22]  Peter Grünwald,et al.  A minimum description length approach to grammar inference , 1995, Learning for Natural Language Processing.