Automatic Selection of High Quality Parses Created By a Fully Unsupervised Parser

The average results obtained by unsupervised statistical parsers have greatly improved in the last few years, but on many specific sentences they are of rather low quality. The output of such parsers is becoming valuable for various applications, and it is radically less expensive to create than manually annotated training data. Hence, automatic selection of high quality parses created by unsupervised parsers is an important problem. In this paper we present PUPA, a POS-based Unsupervised Parse Assessment algorithm. The algorithm assesses the quality of a parse tree using POS sequence statistics collected from a batch of parsed sentences. We evaluate the algorithm by using an unsupervised POS tagger and an unsupervised parser, selecting high quality parsed sentences from English (WSJ) and German (NEGRA) corpora. We show that PUPA outperforms the leading previous parse assessment algorithm for supervised parsers, as well as a strong unsupervised baseline. Consequently, PUPA allows obtaining high quality parses without any human involvement.

[1]  Yoav Seginer,et al.  Fast Unsupervised Incremental Parsing , 2007, ACL.

[2]  Andrew McCallum,et al.  Confidence Estimation for Information Extraction , 2004, NAACL.

[3]  Ari Rappoport,et al.  An Ensemble Method for Selection of High Quality Parses , 2007, ACL.

[4]  Christopher D. Manning,et al.  The unsupervised learning of natural language structure , 2005 .

[5]  Daisuke Kawahara,et al.  Learning Reliability of Parses for Domain Adaptation of Dependency Parsing , 2008, IJCNLP.

[6]  Hitoshi Isahara,et al.  Learning Reliable Information for Dependency Parsing Adaptation , 2008, COLING.

[7]  Dan Klein,et al.  Prototype-Driven Grammar Induction , 2006, ACL.

[8]  Jennifer Chu-Carroll,et al.  In Question Answering, Two Heads Are Better Than One , 2003, NAACL.

[9]  Noah A. Smith,et al.  Annealing Structural Bias in Multilingual Weighted Grammar Induction , 2006, ACL.

[10]  Sanda M. Harabagiu,et al.  COGEX: A Logic Prover for Question Answering , 2003, NAACL.

[11]  Dan Klein,et al.  A Generative Constituent-Context Model for Improved Grammar Induction , 2002, ACL.

[12]  Michael Collins,et al.  Head-Driven Statistical Models for Natural Language Parsing , 2003, CL.

[13]  Jun'ichi Tsujii,et al.  Dependency Parsing and Domain Adaptation with LR Models and Parser Ensembles , 2007, EMNLP.

[14]  Myoung-Wan Koo,et al.  Speech recognition and utterance verification based on a generalized confidence score , 2001, IEEE Trans. Speech Audio Process..

[15]  Ronen Feldman,et al.  Using Corpus Statistics on Entities to Improve Semi-supervised Relation Extraction from the Web , 2007, ACL.

[16]  Philipp Koehn,et al.  Enriching Morphologically Poor Languages for Statistical Machine Translation , 2008, ACL.

[17]  Dan Roth,et al.  The Importance of Syntactic Parsing and Inference in Semantic Role Labeling , 2008, CL.

[18]  Hermann Ney,et al.  Word-Level Confidence Estimation for Machine Translation , 2007, CL.

[19]  Oren Etzioni,et al.  Detecting Parser Errors Using Web-based Semantic Filters , 2006, EMNLP.

[20]  Simon Dennis,et al.  An exemplar-based approach to unsupervised parsing , 2005 .

[21]  Rens Bod,et al.  An All-Subtrees Approach to Unsupervised Parsing , 2006, ACL.

[22]  Ari Rappoport,et al.  Unsupervised Induction of Labeled Parse Trees by Clustering with Syntactic Features , 2008, COLING.

[23]  Eugene Charniak,et al.  Self-Training for Biomedical Parsing , 2008, ACL.

[24]  Kevin Knight,et al.  A Syntax-based Statistical Translation Model , 2001, ACL.

[25]  Rens Bod,et al.  Unsupervised Parsing with U-DOP , 2006, CoNLL.

[26]  Alexander Clark,et al.  Combining Distributional and Morphological Information for Part of Speech Induction , 2003, EACL.

[27]  Dan Roth,et al.  Extraction of Entailed Semantic Relations Through Syntax-Based Comma Resolution , 2008, ACL.

[28]  Feng Lin,et al.  Computing Confidence Scores for All Sub Parse Trees , 2008, ACL.

[29]  Bernard E. M. Jones.  Towards a Syntactic Account of Punctuation , 1996, COLING.

[30]  Rich Caruana,et al.  An empirical comparison of supervised learning algorithms , 2006, ICML.

[31]  Oren Etzioni,et al.  Scaling question answering to the Web , 2001, WWW '01.

[32]  Kevin Knight,et al.  Automatic Prediction of Parser Accuracy , 2008, EMNLP.

[33]  Dan Klein,et al.  Corpus-Based Induction of Syntactic Structure: Models of Dependency and Constituency , 2004, ACL.