Learning Type-Driven Tensor-Based Meaning Representations

This paper investigates the learning of 3rd-order tensors representing the semantics of transitive verbs. The meaning representations are part of a type-driven tensor-based semantic framework, from the newly emerging field of compositional distributional semantics. Standard techniques from the neural networks literature are used to learn the tensors, which are tested on a selectional preference-style task with a simple 2-dimensional sentence space. Promising results are obtained against a competitive corpus-based baseline. We argue that extending this work beyond transitive verbs, and to higher-dimensional sentence spaces, is an interesting and challenging problem for the machine learning community to consider.

[1]  Joshua B. Tenenbaum,et al.  Modelling Relational Data using Bayesian Clustered Tensor Factorization , 2009, NIPS.

[2]  David R. Dowty,et al.  Introduction to Montague semantics , 1980 .

[3]  Thierry Poibeau,et al.  Multi-way Tensor Factorization for Unsupervised Lexical Acquisition , 2012, COLING.

[4]  Geoffrey E. Hinton,et al.  Factored 3-Way Restricted Boltzmann Machines For Modeling Natural Images , 2010, AISTATS.

[5]  Edward Grefenstette,et al.  Category-theoretic quantitative compositional distributional models of natural language semantics , 2013, ArXiv.

[6]  Nathanael Chambers,et al.  Improving the Use of Pseudo-Words for Evaluating Selectional Preferences , 2010, ACL.

[7]  Hinrich Schütze,et al.  Automatic Word Sense Discrimination , 1998, Comput. Linguistics.

[8]  Andrew Y. Ng,et al.  Semantic Compositionality through Recursive Matrix-Vector Spaces , 2012, EMNLP.

[9]  Stephen Clark,et al.  Mathematical Foundations for a Compositional Distributional Model of Meaning , 2010, ArXiv.

[10]  Mirella Lapata,et al.  Vector-based Models of Semantic Composition , 2008, ACL.

[11]  Tom M. Mitchell,et al.  Vector Space Semantic Parsing: A Framework for Compositional Vector Space Models , 2013, CVSM@ACL.

[12]  Marco Baroni,et al.  Nouns are Vectors, Adjectives are Matrices: Representing Adjective-Noun Constructions in Semantic Space , 2010, EMNLP.

[13]  Bing Liu,et al.  Learning with Positive and Unlabeled Examples Using Weighted Logistic Regression , 2003, ICML.

[14]  Mark Steedman,et al.  The syntactic process , 2004, Language, speech, and communication.

[15]  Ehud Rivlin,et al.  Placing search in context: the concept revisited , 2002, TOIS.

[16]  Daoud Clarke,et al.  A Context-Theoretic Framework for Compositionality in Distributional Semantics , 2011, Computational Linguistics.

[17]  James R. Curran,et al.  Wide-Coverage Efficient Statistical Parsing with CCG and Log-Linear Models , 2007, Computational Linguistics.

[18]  Stephen Clark.  Type-Driven Syntax and Semantics for Composing Meaning Vectors , 2013, Quantum Physics and Linguistics.

[19]  AlpaydinEthem,et al.  Cost-Conscious Comparison of Supervised Learning Algorithms over Multiple Data Sets , 2008 .

[20]  John A. Carroll,et al.  Applied morphological processing of English , 2001, Natural Language Engineering.

[21]  Patrick Pantel,et al.  From Frequency to Meaning: Vector Space Models of Semantics , 2010, J. Artif. Intell. Res..

[22]  T. Landauer,et al.  Indexing by Latent Semantic Analysis , 1990 .

[23]  Susan T. Dumais,et al.  Richard Harshman Indexing by Latent Semantic Analysis , 1990 .

[24]  Amy Beth Warriner,et al.  Concreteness ratings for 40 thousand generally known English word lemmas , 2014, Behavior research methods.

[25]  Diarmuid Ó Séaghdha.  Latent Variable Models of Selectional Preference , 2010, ACL.

[26]  Gemma Boleda,et al.  Distributional Semantics in Technicolor , 2012, ACL.

[27]  Stephen Clark,et al.  Improving Distributional Semantic Vectors through Context Selection and Normalisation , 2014, EACL.

[28]  Mark Steedman,et al.  CCGbank: A Corpus of CCG Derivations and Dependency Structures Extracted from the Penn Treebank , 2007, CL.

[29]  Mehrnoosh Sadrzadeh,et al.  Experimental Support for a Categorical Compositional Distributional Model of Meaning , 2011, EMNLP.

[30]  Yoram Singer,et al.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization , 2011, J. Mach. Learn. Res..

[31]  Yoav Goldberg,et al.  A Dataset of Syntactic-Ngrams over Time from a Very Large Corpus of English Books , 2013, *SEMEVAL.