Extracting & Learning a Dependency-Enhanced Type Lexicon for Dutch Submitted by Konstantinos Kogkalidis

This thesis is concerned with type-logical grammars and their practical applicability as tools of reasoning about sentence syntax and semantics. The focal point is narrowed to Dutch, a language exhibiting a large degree of word order variability. In order to overcome difficulties arising as a result of that variability, the thesis explores and expands upon a type grammar based on Multiplicative Intuitionistic Linear Logic, agnostic to word order but enriched with decorations that aim to reduce its proof-theoretic complexity. An algorithm for the conversion of dependency-annotated sentences into type sequences is then implemented, populating the type logic with concrete, data-driven lexical types. Two experiments are ran on the resulting grammar instantiation. The first pertains to the learnability of the type-assignment process by a neural architecture. A novel application of a self-attentive sequence transduction model is proposed; contrary to established practices, it constructs types inductively by internalizing the type-formation syntax, thus exhibiting generalizability beyond a pre-specified type vocabulary. The second revolves around a deductive parsing system that can resolve structural ambiguities by consulting both word and type information; preliminary results suggest both excellent computational efficiency and performance.

[1]  J. Lambek.  The Mathematics of Sentence Structure , 1958 .

[2]  Joachim Lambek,et al.  On the Calculus of Syntactic Types , 1961 .

[3]  C. L. Hamblin.  Translation to and from Polish Notation , 1962, Comput. J..

[4]  Johan van Benthem,et al.  The semantics of variety in categorial grammar , 1988 .

[5]  Johan van Benthem,et al.  Language in action , 1991, J. Philos. Log..

[6]  Philip Gage,et al.  A new algorithm for data compression , 1994 .

[7]  M. Moortgat,et al.  Structural control , 1997 .

[8]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[9]  Michael Moortgat,et al.  Syntactic Annotation for the Spoken Dutch Corpus Project (CGN) , 2000, CLIN.

[10]  Reinhard Muskens,et al.  Lambda grammars and the syntax-semantics interface , 2001 .

[11]  Jürgen Schmidhuber,et al.  LSTM recurrent networks learn simple context-free and context-sensitive languages , 2001, IEEE Trans. Neural Networks.

[12]  Michael Moortgat,et al.  Using the Spoken Dutch Corpus for type-logical grammar induction , 2002, LREC.

[13]  Julia Hockenmaier,et al.  Creating a CCGbank and a Wide-Coverage CCG Lexicon for German , 2006, ACL.

[14]  Gertjan van Noord,et al.  Syntactic Annotation of Large Corpora in STEVIN , 2006, LREC.

[15]  Mark Steedman,et al.  CCGbank: A Corpus of CCG Derivations and Dependency Structures Extracted from the Penn Treebank , 2007, CL.

[16]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[17]  Richard Moot,et al.  Extraction of Type-Logical Supertags from the Spoken Dutch Corpus , 2010 .

[18]  Jason Baldridge,et al.  Non-Transformational Syntax: Formal and Explicit Models of Grammar , 2011 .

[19]  Nitish Srivastava,et al.  Improving neural networks by preventing co-adaptation of feature detectors , 2012, ArXiv.

[20]  Richard Moot,et al.  The Logic of Categorial Grammars , 2012, Lecture Notes in Computer Science.

[21]  Alex Graves,et al.  Supervised Sequence Labelling , 2012 .

[22]  Hideki Mima,et al.  Integrating Multiple Dependency Corpora for Inducing Wide-coverage Japanese CCG Resources , 2013, ACL.

[23]  Ashish Vaswani,et al.  Supertagging With LSTMs , 2016, NAACL.

[24]  Rico Sennrich,et al.  Neural Machine Translation of Rare Words with Subword Units , 2015, ACL.

[25]  Kevin Gimpel,et al.  Bridging Nonlinearities and Stochastic Regularizers with Gaussian Error Linear Units , 2016, ArXiv.

[26]  Luke S. Zettlemoyer,et al.  LSTM CCG Parsing , 2016, NAACL.

[27]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[28]  Jungo Kasai,et al.  TAG Parsing with Neural Networks and Vector Representations of Supertags , 2017, EMNLP.

[29]  Frank Hutter,et al.  Fixing Weight Decay Regularization in Adam , 2017, ArXiv.

[30]  Yue Zhang,et al.  In-Order Transition-based Constituent Parsing , 2017, TACL.

[31]  Quoc V. Le,et al.  Unsupervised Pretraining for Sequence to Sequence Learning , 2016, EMNLP.

[32]  Luke S. Zettlemoyer,et al.  Deep Contextualized Word Representations , 2018, NAACL.

[33]  Yasuhiro Fujiwara,et al.  Sigsoftmax: Reanalysis of the Softmax Bottleneck , 2018, NeurIPS.

[34]  Ilya Sutskever,et al.  Language Models are Unsupervised Multitask Learners , 2019 .

[35]  Michael Moortgat,et al.  Constructive Type-Logical Supertagging With Self-Attention Networks , 2019, RepL4NLP@ACL.