A Comparative Analysis of Knowledge-Intensive and Data-Intensive Semantic Parsers

We present a phenomenon-oriented comparative analysis of the two dominant approaches in task-independent semantic parsing: classic, knowledge-intensive and neural, data-intensive models. To reflect state-of-the-art neural NLP technologies, we introduce a new target structure-centric parser that can produce semantic graphs much more accurately than previous data-driven parsers. We then show that, in spite of comparable performance overall, knowledge- and data-intensive models produce different types of errors, in a way that can be explained by their theoretical properties. This analysis leads to new directions for parser development.

[1]  Xuanjing Huang,et al.  Analyzing Linguistic Knowledge in Sequential Model of Sentence , 2016, EMNLP.

[2]  Mark Steedman,et al.  Surface structure and interpretation , 1996, Linguistic inquiry.

[3]  Stephan Oepen,et al.  WikiWoods: Syntacto-Semantic Annotation for English Wikipedia , 2010, LREC.

[4]  Martin Hilpert,et al.  Construction Grammar and its Application to English , 2014 .

[5]  Fei-Fei Li,et al.  Visualizing and Understanding Recurrent Networks , 2015, ArXiv.

[6]  Johan Bos,et al.  The Groningen Meaning Bank , 2013, JSSP.

[7]  Adele E. Goldberg.  Constructions: a new theoretical approach to language , 2003, Trends in Cognitive Sciences.

[8]  Weiwei Sun,et al.  Accurate SHRG-Based Semantic Parsing , 2018, ACL.

[9]  Peter Jonsson,et al.  Parsing to Noncrossing Dependency Graphs , 2015, Transactions of the Association for Computational Linguistics.

[10]  FlickingerDan.  On building a more efficient grammar by exploiting types , 2000 .

[11]  Weiwei Sun,et al.  Transition-Based Parsing for Deep Dependency Structures , 2016, CL.

[12]  Ilya Sutskever,et al.  Learning to Generate Reviews and Discovering Sentiment , 2017, ArXiv.

[13]  G. Heigold,et al.  A Linguistic Evaluation of Rule-Based, Phrase-Based, and Neural MT Engines , 2017, Prague Bull. Math. Linguistics.

[14]  Yejin Choi,et al.  Neural AMR: Sequence-to-Sequence Models for Parsing and Generation , 2017, ACL.

[15]  Yonatan Belinkov,et al.  Analysis Methods in Neural Language Processing: A Survey , 2018, TACL.

[16]  Emmanuel Dupoux,et al.  Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies , 2016, TACL.

[17]  Stephan Oepen,et al.  Discriminant-Based MRS Banking , 2006, LREC.

[18]  Timothy Dozat,et al.  Simpler but More Accurate Semantic Dependency Parsing , 2018, ACL.

[19]  Alec Radford,et al.  Improving Language Understanding by Generative Pre-Training , 2018 .

[20]  Philipp Koehn,et al.  Abstract Meaning Representation for Sembanking , 2013, LAW@ACL.

[21]  Ari Rappoport,et al.  Universal Conceptual Cognitive Annotation (UCCA) , 2013, ACL.

[22]  Mark Steedman,et al.  Building Deep Dependency Structures using a Wide-Coverage CCG Parser , 2002, ACL.

[23]  C. Fillmore,et al.  Grammatical constructions and linguistic generalizations: The What's X doing Y? construction , 1999 .

[24]  Stephan Oepen,et al.  Towards a Catalogue of Linguistic Graph Banks , 2016, Computational Linguistics.

[25]  Ari Rappoport,et al.  A Transition-Based Directed Acyclic Graph Parser for UCCA , 2017, ACL.

[26]  Weiwei Sun,et al.  A Data-Driven, Factorization Parser for CCG Dependency Structures , 2015, ACL.

[27]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[28]  Weiwei Sun,et al.  Parsing to 1-Endpoint-Crossing, Pagenumber-2 Graphs , 2017, ACL.

[29]  Vilmos Ágel,et al.  Dependency Grammar and Valency Theory , 2009 .

[30]  Phil Blunsom,et al.  Robust Incremental Neural Semantic Graph Parsing , 2017, ACL.

[31]  Roland Kuhn,et al.  A Challenge Set for French -> English Machine Translation , 2018, ArXiv.

[32]  Stephan Oepen,et al.  Parser Evaluation Using Elementary Dependency Matching , 2011, IWPT.

[33]  Berthold Crysmann,et al.  Some Fine Points of Hybrid Natural Language Parsing , 2008, LREC.

[34]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[35]  Johan Bos,et al.  Developing a large semantically annotated corpus , 2012, LREC.

[36]  Omer Levy,et al.  Deep RNNs Encode Soft Hierarchical Syntax , 2018, ACL.

[37]  Alexander Kuhnle,et al.  Deep learning evaluation using deep linguistic processing , 2017, ArXiv.

[38]  Laura A. Michaelis,et al.  TOWARD A CONSTRUCTION-BASED THEORY OF LANGUAGE FUNCTION : THE CASE OF NOMINAL EXTRAPOSITION , 1996 .

[39]  Zi Lin,et al.  Parsing Meaning Representations: Is Easier Always Better? , 2019, Proceedings of the First International Workshop on Designing Meaning Representations.

[40]  Eliyahu Kiperwasser,et al.  Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations , 2016, TACL.

[41]  Ivan A. Sag,et al.  Book Reviews: Head-driven Phrase Structure Grammar and German in Head-driven Phrase-structure Grammar , 1996, CL.

[42]  Yonatan Belinkov,et al.  Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks , 2017, IJCNLP.

[43]  Emily M. Bender,et al.  Towards an Encyclopedia of Compositional Semantics: Documenting the Interface of the English Resource Grammar , 2014, LREC.

[44]  Weiwei Sun,et al.  Peking at MRP 2019: Factorization- and Composition-Based Parsing for Elementary Dependency Structures , 2019, CoNLL.

[45]  Pierre Isabelle,et al.  A Challenge Set Approach to Evaluating Machine Translation , 2017, EMNLP.

[46]  Xiaochang Peng,et al.  A Synchronous Hyperedge Replacement Grammar based approach for AMR parsing , 2015, CoNLL.

[47]  Michael E. Phelps,et al.  PET , 2004, Springer New York.

[48]  松本 裕治,et al.  Abstract Meaning Representationを用いた名詞句の意味構造解析 , 2015 .

[49]  C. Fillmore,et al.  Regularity and Idiomaticity in Grammatical Constructions: The Case of Let Alone , 1988 .

[50]  Alex Wang,et al.  What do you learn from context? Probing for sentence structure in contextualized word representations , 2019, ICLR.

[51]  Samuel R. Bowman,et al.  Neural Network Acceptability Judgments , 2018, Transactions of the Association for Computational Linguistics.

[52]  Dan Flickinger,et al.  An Open Source Grammar Development Environment and Broad-coverage English Grammar Using HPSG , 2000, LREC.

[53]  Lucien Tesnière,et al.  Elements of Structural Syntax , 2015 .

[54]  Graham Neubig,et al.  On-the-fly Operation Batching in Dynamic Computation Graphs , 2017, NIPS.

[55]  Noah A. Smith,et al.  Deep Multitask Learning for Semantic Dependency Parsing , 2017, ACL.

[56]  Min Zhang,et al.  SUDA-Alibaba at MRP 2019: Graph-Based Models with BERT , 2019, CoNLL Shared Task.

[57]  José A. R. Fonollosa,et al.  Character-based Neural Machine Translation , 2016, ACL.

[58]  Dan Flickinger,et al.  Minimal Recursion Semantics: An Introduction , 2005 .

[59]  Omer Levy,et al.  GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding , 2018, BlackboxNLP@EMNLP.

[60]  Stephan Oepen,et al.  Efficiency in Unification-Based N-Best Parsing , 2007, Trends in Parsing Technology.

[61]  Mark Steedman,et al.  Wide-Coverage Semantic Representations from a CCG Parser , 2004, COLING.

[62]  Kevin Knight,et al.  Smatch: an Evaluation Metric for Semantic Feature Structures , 2013, ACL.

[63]  Jun'ichi Tsujii,et al.  Feature Forest Models for Probabilistic HPSG Parsing , 2008, CL.

[64]  Emily M. Bender,et al.  Parser Evaluation over Local and Non-Local Deep Dependencies in a Large Corpus , 2011, EMNLP.

[65]  James R. Curran,et al.  Wide-Coverage Efficient Statistical Parsing with CCG and Log-Linear Models , 2007, Computational Linguistics.

[66]  Martin Kay,et al.  Syntactic Process , 1979, ACL.

[67]  Richard Socher,et al.  Learned in Translation: Contextualized Word Vectors , 2017, NIPS.

[68]  Xing Shi,et al.  Does String-Based Neural MT Learn Source Syntax? , 2016, EMNLP.

[69]  Mark Johnson,et al.  AMR dependency parsing with a typed semantic algebra , 2018, ACL.

[70]  Luke S. Zettlemoyer,et al.  Dissecting Contextual Word Embeddings: Architecture and Representation , 2018, EMNLP.

[71]  Ulrich Callmeier,et al.  PET – a platform for experimentation with efficient HPSG processing techniques , 2000, Natural Language Engineering.

[72]  Franziska Frankfurter,et al.  Constructions: A construction grammar approach to argument structure: Adele E. Goldberg, Chicago, IL: The University of Chicago Press, 1995. xi + 265 pp , 1998 .

[73]  Luke S. Zettlemoyer,et al.  Broad-coverage CCG Semantic Parsing with AMR , 2015, EMNLP.

[74]  Stephan Oepen,et al.  Who Did What to Whom? A Contrastive Study of Syntacto-Semantic Dependencies , 2012, LAW@ACL.

[75]  Rico Sennrich,et al.  How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairs , 2016, EACL.

[76]  Stephan Oepen,et al.  SemEval 2014 Task 8: Broad-Coverage Semantic Dependency Parsing , 2014, *SEMEVAL.

[77]  Yonatan Belinkov,et al.  Identifying and Controlling Important Neurons in Neural Machine Translation , 2018, ICLR.

[78]  Luke S. Zettlemoyer,et al.  Deep Contextualized Word Representations , 2018, NAACL.

[79]  Jaime G. Carbonell,et al.  A Discriminative Graph-Based Parser for the Abstract Meaning Representation , 2014, ACL.

[80]  Ann A. Copestake,et al.  Invited Talk: Slacker Semantics: Why Superficiality, Dependency and Avoidance of Commitment can be the Right Way to Go , 2009, EACL.