Dependency Grammar Induction with Neural Lexicalization and Big Training Data

We study the impact of big models (in terms of the degree of lexicalization) and big data (in terms of the training corpus size) on dependency grammar induction. We experimented with L-DMV, a lexicalized version of Dependency Model with Valence and L-NDMV, our lexicalized extension of the Neural Dependency Model with Valence. We find that L-DMV only benefits from very small degrees of lexicalization and moderate sizes of training corpora. L-NDMV can benefit from big training data and lexicalization of greater degrees, especially when enhanced with good model initialization, and it achieves a result that is competitive with the current state-of-the-art.

[1]  Dan Klein,et al.  Corpus-Based Induction of Syntactic Structure: Models of Dependency and Constituency , 2004, ACL.

[2]  Kewei Tu,et al.  Unsupervised Neural Dependency Parsing , 2016, EMNLP.

[3]  Dan Klein,et al.  Online EM for Unsupervised Models , 2009, NAACL.

[4]  Mark Johnson,et al.  Using Universal Linguistic Knowledge to Guide Grammar Induction , 2010, EMNLP.

[5]  John DeNero,et al.  Painless Unsupervised Learning with Features , 2010, NAACL.

[6]  Ben Taskar,et al.  Sparsity in Dependency Grammar Induction , 2010, ACL.

[7]  Mark Johnson,et al.  Improving Unsupervised Dependency Parsing with Richer Contexts and Smoothing , 2009, NAACL.

[8]  Mark Johnson,et al.  Grammar induction from (lots of) words alone , 2016, COLING.

[9]  Noah A. Smith,et al.  Logistic Normal Priors for Unsupervised Probabilistic Grammar Induction , 2008, NIPS.

[10]  Valentin I. Spitkovsky,et al.  Breaking Out of Local Optima with Count Transforms and Model Recombination: A Study in Grammar Induction , 2013, EMNLP.

[11]  Kewei Tu,et al.  Unambiguity Regularization for Unsupervised Learning of Probabilistic Grammars , 2012, EMNLP.

[12]  Phil Blunsom,et al.  Unsupervised Induction of Tree Substitution Grammars for Dependency Parsing , 2010, EMNLP.

[13]  Milan Straka,et al.  Stop-probability estimates computed on a large corpus improve Unsupervised Dependency Parsing , 2013, ACL.

[14]  Joakim Nivre,et al.  Inductive Dependency Parsing , 2006, Text, speech and language technology.

[15]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[16]  Dan Klein,et al.  Feature-Rich Part-of-Speech Tagging with a Cyclic Dependency Network , 2003, NAACL.

[17]  Phong Le,et al.  Unsupervised Dependency Parsing: Let’s Use Supervised Parsers , 2015, NAACL.

[18]  Petr Sojka,et al.  Software Framework for Topic Modelling with Large Corpora , 2010 .