Embedding Syntax and Semantics of Prepositions via Tensor Decomposition

Prepositions are among the most frequent words in English and play complex roles in the syntax and semantics of sentences. Not surprisingly, they pose well-known difficulties in automatic processing of sentences (prepositional attachment ambiguities and idiosyncratic uses in phrases). Existing methods on preposition representation treat prepositions no different from content words (e.g., word2vec and GloVe). In addition, recent studies aiming at solving prepositional attachment and preposition selection problems depend heavily on external linguistic resources and use dataset-specific word representations. In this paper we use word-triple counts (one of the triples being a preposition) to capture a preposition's interaction with its attachment and complement. We then derive preposition embeddings via tensor decomposition on a large unlabeled corpus. We reveal a new geometry involving Hadamard products and empirically demonstrate its utility in paraphrasing phrasal verbs. Furthermore, our preposition embeddings are used as simple features in two challenging downstream tasks: preposition selection and prepositional attachment disambiguation. We achieve results comparable to or better than the state-of-the-art on multiple standardized datasets.

[1]  Anima Anandkumar,et al.  Guaranteed Non-Orthogonal Tensor Decomposition via Alternating Rank-1 Updates , 2014, ArXiv.

[2]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[3]  Martin Grund,et al.  Correct Me If I'm Wrong: Fixing Grammatical Errors by Preposition Ranking , 2014, CIKM.

[4]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[5]  Yang Xiang,et al.  A Hybrid Model For Grammatical Error Correction , 2013, CoNLL Shared Task.

[6]  Dirk Hovy,et al.  What’s in a Preposition? Dimensions of Sense Disambiguation for an Interesting Word Class , 2010, COLING.

[7]  R. Huddleston.  Introduction to the Grammar of English: Verbs, nouns and adjectives: the boundaries between them , 1984 .

[8]  Dan Klein,et al.  Parser Showdown at the Wall Street Corral: An Empirical Investigation of Error Types in Parser Output , 2012, EMNLP.

[9]  Percy Liang,et al.  From Language to Programs: Bridging Reinforcement Learning and Maximum Marginal Likelihood , 2017, ACL.

[10]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[11]  Vatsal Sharan,et al.  Orthogonalized ALS: A Theoretically Principled Tensor Decomposition Algorithm for Practical Use , 2017, ICML.

[12]  Steven Skiena,et al.  Polyglot: Distributed Word Representations for Multilingual NLP , 2013, CoNLL.

[13]  Chris Dyer,et al.  Ontology-Aware Token Embeddings for Prepositional Phrase Attachment , 2017, ACL.

[14]  Eric Brill,et al.  A Rule-Based Approach to Prepositional Phrase Attachment Disambiguation , 1994, COLING.

[15]  Eliyahu Kiperwasser,et al.  Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations , 2016, TACL.

[16]  Percy Liang,et al.  Tensor Factorization via Matrix Factorization , 2015, AISTATS.

[17]  Erhard W. Hinrichs,et al.  Transition-based dependency parsing with topological fields , 2016, ACL.

[18]  Yu-Wei Chang,et al.  CoNLL-2013 Shared Task: Grammatical Error Correction NTHU System Description , 2013, CoNLL Shared Task.

[19]  Won-Sook Lee,et al.  Disambiguating Spatial Prepositions Using Deep Convolutional Networks , 2017, AAAI.

[20]  Mark Dredze,et al.  Embedding Lexical Features via Low-Rank Tensors , 2016, HLT-NAACL.

[21]  Jeanette Speer DeCarrico.  The Structure of English: Studies in Form and Function for Language Teaching , 2000 .

[22]  Hinrich Schütze,et al.  AutoExtend: Extending Word Embeddings to Embeddings for Synsets and Lexemes , 2015, ACL.

[23]  Jingwei Zhang,et al.  Word Semantic Representations using Bayesian Probabilistic Tensor Factorization , 2014, EMNLP.

[24]  Jason Weston,et al.  Reading Wikipedia to Answer Open-Domain Questions , 2017, ACL.

[25]  Jorge J. Moré,et al.  The Levenberg-Marquardt algo-rithm: Implementation and theory , 1977 .

[26]  P. Comon,et al.  Tensor decompositions, alternating least squares and other tales , 2009 .

[27]  Claudia Leacock,et al.  Automated Grammatical Error Correction for Language Learners , 2010, COLING.

[28]  Raymond Hendy Susanto,et al.  The CoNLL-2014 Shared Task on Grammatical Error Correction , 2014 .

[29]  Yonatan Belinkov,et al.  Exploring Compositional Architectures and Word Vector Representations for Prepositional Phrase Attachment , 2014, Transactions of the Association for Computational Linguistics.

[30]  Dan Roth,et al.  The University of Illinois System in the CoNLL-2013 Shared Task , 2013, CoNLL Shared Task.

[31]  Jorge Nocedal,et al.  On the limited memory BFGS method for large scale optimization , 1989, Math. Program..

[32]  Nathan Schneider,et al.  A Hierarchy with, of, and for Preposition Supersenses , 2015, LAW@NAACL-HLT.