Generative Incremental Dependency Parsing with Neural Networks

We propose a neural network model for scalable generative transition-based dependency parsing. A probability distribution over both sentences and transition sequences is parameterised by a feedforward neural network. The model surpasses the accuracy and speed of previous generative dependency parsers, reaching 91.1% UAS. Perplexity results show a strong improvement over n-gram language models, opening the way to the efficient integration of syntax into neural models for language generation.

[1]  Danqi Chen,et al.  A Fast and Accurate Dependency Parser using Neural Networks , 2014, EMNLP.

[2]  Richard M. Schwartz,et al.  Fast and Robust Neural Network Joint Models for Statistical Machine Translation , 2014, ACL.

[3]  Baltescu Paul,et al.  OxLM: A Neural Language Modelling Framework for Machine Translation , 2014, Prague Bull. Math. Linguistics.

[4]  Richard Johansson,et al.  Extended Constituent-to-Dependency Conversion for English , 2007, NODALIDA.

[5]  Joakim Nivre,et al.  Deterministic Dependency Parsing of English Text , 2004, COLING.

[6]  Dan Klein,et al.  Feature-Rich Part-of-Speech Tagging with a Cyclic Dependency Network , 2003, NAACL.

[7]  Frederick Jelinek,et al.  Structured language modeling , 2000, Comput. Speech Lang..

[8]  Ivan Titov,et al.  A Latent Variable Model for Generative Dependency Parsing , 2007, Trends in Parsing Technology.

[9]  Yoshua Bengio,et al.  A Neural Probabilistic Language Model , 2003, J. Mach. Learn. Res..

[10]  Phil Blunsom,et al.  Compositional Morphology for Word Representations and Language Modelling , 2014, ICML.

[11]  Robert L. Mercer,et al.  Class-Based n-gram Models of Natural Language , 1992, CL.

[12]  Andrew McCallum,et al.  Bayesian Modeling of Dependency Trees Using Hierarchical Pitman-Yor Priors , 2008 .

[13]  Hermann Ney,et al.  Improved backing-off for M-gram language modeling , 1995, 1995 International Conference on Acoustics, Speech, and Signal Processing.

[14]  Phil Blunsom,et al.  A Bayesian Model for Generative Transition-based Dependency Parsing , 2015, DepLing.

[15]  Brian Roark,et al.  Probabilistic Top-Down Parsing and Language Modeling , 2001, CL.

[16]  Jason Eisner,et al.  Three New Probabilistic Models for Dependency Parsing: An Exploration , 1996, COLING.

[17]  Yoram Singer,et al.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization , 2011, J. Mach. Learn. Res..

[18]  Thorsten Brants,et al.  One billion word benchmark for measuring progress in statistical language modeling , 2013, INTERSPEECH.

[19]  Phil Blunsom,et al.  Recurrent Continuous Translation Models , 2013, EMNLP.

[20]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[21]  Dan Klein,et al.  Accurate Unlexicalized Parsing , 2003, ACL.

[22]  Phil Blunsom,et al.  Pragmatic Neural Language Modelling in Machine Translation , 2014, NAACL.

[23]  NivreJoakim.  Algorithms for deterministic incremental dependency parsing , 2008 .

[24]  Noah A. Smith,et al.  Transition-Based Dependency Parsing with Stack Long Short-Term Memory , 2015, ACL.

[25]  Lukás Burget,et al.  Recurrent neural network based language model , 2010, INTERSPEECH.

[26]  Kevin Gimpel,et al.  Tailoring Continuous Word Representations for Dependency Parsing , 2014, ACL.

[27]  Geoffrey E. Hinton,et al.  Three new graphical models for statistical language modelling , 2007, ICML '07.

[28]  Beatrice Santorini,et al.  Building a Large Annotated Corpus of English: The Penn Treebank , 1993, CL.

[29]  Stephen Clark,et al.  A Tale of Two Parsers: Investigating and Combining Graph-based and Transition-based Dependency Parsing , 2008, EMNLP.

[30]  Joakim Nivre,et al.  A Transition-Based System for Joint Part-of-Speech Tagging and Labeled Non-Projective Dependency Parsing , 2012, EMNLP.

[31]  Christopher D. Manning,et al.  The Stanford Typed Dependencies Representation , 2008, CF+CDPE@COLING.

[32]  Timothy J. Robinson,et al.  Sequential Monte Carlo Methods in Practice , 2003 .

[33]  Ahmad Emami,et al.  A Neural Syntactic Language Model , 2005, Machine Learning.

[34]  Phil Blunsom,et al.  OxLM: A Neural Language Modelling Framework for Machine Translation , 2014, Prague Bull. Math. Linguistics.

[35]  Andrew McCallum,et al.  Transition-based Dependency Parsing with Selectional Branching , 2013, ACL.

[36]  Slav Petrov,et al.  Structured Training for Neural Network Transition-Based Parsing , 2015, ACL.

[37]  Omer Levy,et al.  Dependency-Based Word Embeddings , 2014, ACL.

[38]  Eugene Charniak,et al.  Immediate-Head Parsing for Language Models , 2001, ACL.

[39]  Giorgio Satta,et al.  Exact Inference for Generative Probabilistic Non-Projective Dependency Parsing , 2011, EMNLP.

[40]  Joakim Nivre,et al.  Transition-based Dependency Parsing with Rich Non-local Features , 2011, ACL.