The Effect of Dependency Representation Scheme on Syntactic Language Modelling

There has been considerable work on syntactic language models and they have advanced greatly over the last decade. Most of them have used a probabilistic contextfree grammar (PCFG) or a dependency grammar (DG). In particular, DG has attracted more and more interest in the past years since dependency parsing has achieved great success. While much work has evaluated the effects of different dependency representations in the context of parsing, there has been relatively little investigation into them on a syntactic language model. In this work, we conduct the first assessment of three dependency representations on a transition-based dependency parsing language model. We show that the choice of dependency representation has an impact on overall performance from the perspective of language modelling.

[1]  Andreas Stolcke,et al.  SRILM - an extensible language modeling toolkit , 2002, INTERSPEECH.

[2]  Stephan Oepen,et al.  Survey on parsing three dependency representations for English , 2013, ACL.

[3]  Richárd Farkas,et al.  Stacking of Dependency and Phrase Structure Parsers , 2012, COLING.

[4]  Rebecca Jonson.  Generating Statistical Language Models from Interpretation Grammars in Dialogue Systems , 2006, EACL.

[5]  F ChenStanley,et al.  An Empirical Study of Smoothing Techniques for Language Modeling , 1996, ACL.

[6]  Mary P. Harper,et al.  The SuperARV Language Model: Investigating the Effectiveness of Tightly Integrating Multiple Knowledge Sources , 2002, EMNLP.

[7]  Jorge Nocedal,et al.  On the limited memory BFGS method for large scale optimization , 1989, Math. Program..

[8]  Alexander M. Fraser,et al.  A Smorgasbord of Features for Statistical Machine Translation , 2004, NAACL.

[9]  Beatrice Santorini,et al.  Building a Large Annotated Corpus of English: The Penn Treebank , 1993, CL.

[10]  Sampo Pyysalo,et al.  Evaluating Dependency Representations for Event Extraction , 2010, COLING.

[11]  Giorgio Satta,et al.  Exact Inference for Generative Probabilistic Non-Projective Dependency Parsing , 2011, EMNLP.

[12]  Frederick Jelinek,et al.  Structured language modeling , 2000, Comput. Speech Lang..

[13]  Daniel Zeman,et al.  Coordination Structures in Dependency Treebanks , 2013, ACL.

[14]  Brian Roark,et al.  Probabilistic Top-Down Parsing and Language Modeling , 2001, CL.

[15]  James R. Curran,et al.  Improving Combinatory Categorial Grammar Parse Reranking with Dependency Grammar Features , 2012, COLING.

[16]  Andreas Vlachos,et al.  Dependency Language Models for Sentence Completion , 2013, EMNLP.

[17]  Noah A. Smith.  Adversarial Evaluation for Models of Natural Language , 2012, ArXiv.

[18]  Geoffrey Zweig,et al.  A Challenge Set for Advancing Language Modeling , 2012, WLM@NAACL-HLT.

[19]  Christopher D. Manning,et al.  The Stanford Typed Dependencies Representation , 2008, CF+CDPE@COLING.

[20]  Adam L. Berger,et al.  A Maximum Entropy Approach to Natural Language Processing , 1996, CL.

[21]  Peng Xu,et al.  Random forests and the data sparseness problem in language modeling , 2007, Comput. Speech Lang..

[22]  R. Rosenfeld,et al.  Two decades of statistical language modeling: where do we go from here? , 2000, Proceedings of the IEEE.

[23]  Eduard H. Hovy,et al.  A Fast, Accurate, Non-Projective, Semantically-Enriched Parser , 2011, EMNLP.

[24]  Martha Palmer,et al.  Robust Constituent-to-Dependency Conversion for English , 2010 .

[25]  Lilja Øvrelid,et al.  Cross-framework parser stacking for data-driven dependency parsing , 2009, TAL.

[26]  Eugene Charniak,et al.  Immediate-Head Parsing for Language Models , 2001, ACL.

[27]  Jun'ichi Tsujii,et al.  HPSG Parsing with Shallow Dependency Constraints , 2007, ACL.

[28]  Joakim Nivre,et al.  Inductive Dependency Parsing , 2006, Text, speech and language technology.

[29]  Joakim Nivre,et al.  Algorithms for Deterministic Incremental Dependency Parsing , 2008, CL.

[30]  Richard Johansson,et al.  Extended Constituent-to-Dependency Conversion for English , 2007, NODALIDA.

[31]  Eugene Charniak,et al.  A TAG-based noisy-channel model of speech repairs , 2004, ACL.

[32]  Sigrid Klerke,et al.  Down-stream effects of tree-to-dependency conversions , 2013, HLT-NAACL.

[33]  Matthew Lease,et al.  Parsing and its applications for conversational speech , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..