Punctuation: Making a Point in Unsupervised Dependency Parsing

We show how punctuation can be used to improve unsupervised dependency parsing. Our linguistic analysis confirms the strong connection between English punctuation and phrase boundaries in the Penn Treebank. However, approaches that naively include punctuation marks in the grammar (as if they were words) do not perform well with Klein and Manning's Dependency Model with Valence (DMV). Instead, we split a sentence at punctuation and impose parsing restrictions over its fragments. Our grammar inducer is trained on the Wall Street Journal (WSJ) and achieves 59.5% accuracy out-of-domain (Brown sentences with 100 or fewer words), more than 6% higher than the previous best results. Further evaluation, using the 2006/7 CoNLL sets, reveals that punctuation aids grammar induction in 17 of 18 languages, for an overall average net gain of 1.3%. Some of this improvement is from training, but more than half is from parsing with induced constraints, in inference. Punctuation-aware decoding works with existing (even already-trained) parsing models and always increased accuracy in our experiments.

[1]  R. Lee Humphreys,et al.  The linguistics of punctuation , 2004, Machine Translation.

[2]  Yoav Seginer,et al.  Fast Unsupervised Incremental Parsing , 2007, ACL.

[3]  Eugene Charniak,et al.  Evaluating Unsupervised Part-of-Speech Tagging for Grammar Induction , 2008, COLING.

[4]  Dan Roth,et al.  Learning and Inference over Constrained Output , 2005, IJCAI.

[5]  Dekang Lin,et al.  Dependency-Based Evaluation of Minipar , 2003 .

[6]  Ji-Hwan Kim,et al.  Implementation of automatic capitalisation generation systems for speech input , 2002, 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[7]  Dilek Z. Hakkani-Tür,et al.  Punctuating speech for information extraction , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[8]  John D. Lafferty,et al.  Cyberpunc: a lightweight punctuation annotation system for speech , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[9]  Ming-Wei Chang,et al.  Guiding Semi-Supervision with Constraint-Driven Learning , 2007, ACL.

[10]  I. D. Melamed.  Measuring Semantic Entropy , 1997 .

[11]  Phil Blunsom,et al.  Unsupervised Induction of Tree Substitution Grammars for Dependency Parsing , 2010, EMNLP.

[12]  Kevin C. Yeh.  Bilingual Sentence Alignment Based on Punctuation Marks , 2003, ROCLING.

[13]  Eugene Charniak,et al.  Parsing and Disfluency Placement , 2002, EMNLP.

[14]  Dilek Z. Hakkani-Tür,et al.  IMPACT OF AUTOMATIC COMMA PREDICTION ON POS/NAME TAGGING OF SPEECH , 2006, 2006 IEEE Spoken Language Technology Workshop.

[15]  Joakim Nivre,et al.  MaltParser: A Language-Independent System for Data-Driven Dependency Parsing , 2007, Natural Language Engineering.

[16]  John DeNero,et al.  Painless Unsupervised Learning with Features , 2010, NAACL.

[17]  Christopher D. Manning,et al.  Joint Parsing and Named Entity Recognition , 2009, NAACL.

[18]  Mark Johnson,et al.  Improving Unsupervised Dependency Parsing with Richer Contexts and Smoothing , 2009, NAACL.

[19]  Daniel Dominic Sleator,et al.  Parsing English with a Link Grammar , 1995, IWPT.

[20]  Hsin-Hsi Chen,et al.  Integrating Punctuation Rules and Naïve Bayesian Model for Chinese Creation Title Recognition , 2005, IJCNLP.

[21]  Noah A. Smith,et al.  Parsing with Soft and Hard Constraints on Dependency Length , 2005 .

[22]  Toshikazu Fukushima,et al.  A Sentence Analysis Method for a Japanese Book Reading Machine for the Blind , 1986, ACL.

[23]  Matthew Lease,et al.  Effective Use of Prosody in Parsing Conversational Speech , 2005, HLT.

[24]  Mark Johnson,et al.  Robust probabilistic predictive syntactic processing: motivations, models, and applications , 2001 .

[25]  Sebastian Riedel,et al.  The CoNLL 2007 Shared Task on Dependency Parsing , 2007, EMNLP.

[26]  Valentin I. Spitkovsky,et al.  Profiting from Mark-Up: Hyper-Text Annotations for Guided Parsing , 2010, ACL.

[27]  Philip J. Hayes,et al.  Flexible Parsing , 1980, ACL.

[28]  Glenn Carroll,et al.  Two Experiments on Learning Probabilistic Dependency Grammars from Corpora , 1992 .

[29]  Maosong Sun,et al.  Punctuation as Implicit Annotations for Chinese Word Segmentation , 2009, CL.

[30]  Christine D. Doran,et al.  Incorporating Punctuation Into the Sentence Grammar: A Lexicalized Tree Adjoining Grammar Perspective , 1998 .

[31]  Michael Collins,et al.  Three Generative, Lexicalised Models for Statistical Parsing , 1997, ACL.

[32]  Eugene Charniak,et al.  A Maximum-Entropy-Inspired Parser , 2000, ANLP.

[33]  Beatrice Santorini,et al.  Building a Large Annotated Corpus of English: The Penn Treebank , 1993, CL.

[34]  Fernando Pereira,et al.  Inside-Outside Reestimation From Partially Bracketed Corpora , 1992, HLT.

[35]  Ted Briscoe,et al.  Parsing (with) Punctuation etc , 1994 .

[36]  Kenji Kita,et al.  Spoken Language Translation System , 1993, IJCAI.

[37]  Mark A. Paskin,et al.  Grammatical Bigrams , 2001, NIPS.

[38]  Ben Taskar,et al.  Posterior Sparsity in Unsupervised Dependency Parsing , 2011, J. Mach. Learn. Res..

[39]  Sabine Buchholz,et al.  CoNLL-X Shared Task on Multilingual Dependency Parsing , 2006, CoNLL.

[40]  William Yang Wang,et al.  “Got You!”: Automatic Vandalism Detection in Wikipedia with Web-based Shallow Syntactic-Semantic Modeling , 2010, COLING.

[41]  Michiel Bacchiani,et al.  Restoring punctuation and capitalization in transcribed speech , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[42]  Michael Collins,et al.  Head-Driven Statistical Models for Natural Language Parsing , 2003, CL.

[43]  Efstathios Stamatatos,et al.  Text Genre Detection Using Common Word Frequencies , 2000, COLING.

[44]  Alexander Clark,et al.  Inducing Syntactic Categories by Context Distribution Clustering , 2000, CoNLL/LLL.

[45]  Valentin I. Spitkovsky,et al.  Viterbi Training Improves Unsupervised Dependency Parsing , 2010, CoNLL.

[46]  Giorgio Satta,et al.  Efficient Parsing for Bilexical Context-Free Grammars and Head Automaton Grammars , 1999, ACL.

[47]  Michael White,et al.  A More Precise Analysis of Punctuation for Broad-Coverage Surface Realization with CCG , 2008, COLING 2008.

[48]  Bernard E. M. Jones.  Exploring The Role Of Punctuation In Parsing Natural Text , 1994, COLING.

[49]  Dan Klein,et al.  Corpus-Based Induction of Syntactic Structure: Models of Dependency and Constituency , 2004, ACL.

[50]  Hermann Ney,et al.  Automatic sentence segmentation and punctuation prediction for spoken language translation , 2006, IWSLT.

[51]  Mark Johnson,et al.  PCFG Models of Linguistic Tree Representations , 1998, CL.

[52]  Chengqing Zong,et al.  A Hierarchical Parsing Approach with Punctuation Processing for Long Chinese Sentences , 2005, IJCNLP.

[53]  J. Pitman,et al.  The two-parameter Poisson-Dirichlet distribution derived from a stable subordinator , 1997 .

[54]  Dan Roth,et al.  Integer linear programming inference for conditional random fields , 2005, ICML.

[55]  Daniel Jurafsky,et al.  Semantic Role Chunking Combining Complementary Syntactic Views , 2005, CoNLL.

[56]  Hwee Tou Ng,et al.  Better Punctuation Prediction with Dynamic Conditional Random Fields , 2010, EMNLP.