Parsing as Language Modeling

We recast syntactic parsing as a language modeling problem and use recent advances in neural network language modeling to achieve a new state of the art for constituency Penn Treebank parsing — 93.8 F1 on section 23, using 2-21 as training, 24 as development, plus tri-training. When trees are converted to Stanford dependencies, UAS and LAS are 95.9% and 94.1%.

[1]  Beatrice Santorini,et al.  Building a Large Annotated Corpus of English: The Penn Treebank , 1993, CL.

[2]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[3]  Eugene Charniak,et al.  A Maximum-Entropy-Inspired Parser , 2000, ANLP.

[4]  Jürgen Schmidhuber,et al.  Learning Precise Timing with LSTM Recurrent Networks , 2003, J. Mach. Learn. Res..

[5]  Yoshua Bengio,et al.  Hierarchical Probabilistic Neural Network Language Model , 2005, AISTATS.

[6]  Eugene Charniak,et al.  Effective Self-Training for Parsing , 2006, NAACL.

[7]  Slav Petrov,et al.  Products of Random Latent Variable Grammars , 2010, NAACL.

[8]  Mary P. Harper,et al.  Self-Training with Products of Latent Variable Grammars , 2010, EMNLP.

[9]  Mohamed Chtourou,et al.  On the training of recurrent neural networks , 2011, Eighth International Multi-Conference on Systems, Signals & Devices.

[10]  Aapo Hyvärinen,et al.  Noise-Contrastive Estimation of Unnormalized Statistical Models, with Applications to Natural Image Statistics , 2012, J. Mach. Learn. Res..

[11]  Hiroyuki Shindo,et al.  Bayesian Symbol-Refined Tree Substitution Grammars for Syntactic Parsing , 2012, ACL.

[12]  Yue Zhang,et al.  Fast and Accurate Shift-Reduce Constituent Parsing , 2013, ACL.

[13]  Razvan Pascanu,et al.  On the difficulty of training recurrent neural networks , 2012, ICML.

[14]  Alex Graves,et al.  Generating Sequences With Recurrent Neural Networks , 2013, ArXiv.

[15]  Geoffrey E. Hinton,et al.  Training Recurrent Neural Networks , 2013 .

[16]  Min Zhang,et al.  Ambiguity-aware Ensemble Training for Semi-supervised Dependency Parsing , 2014, ACL.

[17]  Noah A. Smith,et al.  An Empirical Comparison of Parsing Methods for Stanford Dependencies , 2014, ArXiv.

[18]  Christopher Kermorvant,et al.  Dropout Improves Recurrent Neural Networks for Handwriting Recognition , 2013, 2014 14th International Conference on Frontiers in Handwriting Recognition.

[19]  Wojciech Zaremba,et al.  Recurrent Neural Network Regularization , 2014, ArXiv.

[20]  Geoffrey E. Hinton,et al.  Grammar as a Foreign Language , 2014, NIPS.

[21]  Wojciech Zaremba,et al.  An Empirical Exploration of Recurrent Network Architectures , 2015, ICML.

[22]  Eugene Charniak,et al.  Syntactic Parse Fusion , 2015, EMNLP.

[23]  Yonghui Wu,et al.  Exploring the Limits of Language Modeling , 2016, ArXiv.

[24]  Slav Petrov,et al.  Globally Normalized Transition-Based Neural Networks , 2016, ACL.

[25]  Noah A. Smith,et al.  Recurrent Neural Network Grammars , 2016, NAACL.

[26]  Alexander M. Rush,et al.  Character-Aware Neural Language Models , 2015, AAAI.

[27]  Quoc V. Le,et al.  Multi-task Sequence to Sequence Learning , 2015, ICLR.