Semantics in High-Dimensional Space

Geometric models are used for modelling meaning in various semantic-space models. They are seductive in their simplicity and their imaginative qualities, and for that reason, their metaphorical power risks leading our intuitions astray: human intuition works well in a three-dimensional world but is overwhelmed by higher dimensionalities. This note is intended to warn about some practical pitfalls of using high-dimensional geometric representation as a knowledge representation and a memory model—challenges that can be met by informed design of the representation and its application.

[1]  Hinrich Schütze,et al.  Word Space , 1992, NIPS.

[2]  Ross W. Gayler.  Vector Symbolic Architectures answer Jackendoff's challenges for cognitive neuroscience , 2004, ArXiv.

[3]  Trevor Cohen,et al.  Reasoning with vectors: A continuous model for fast robust inference , 2015, Log. J. IGPL.

[4]  Anders Holst,et al.  Random indexing of text samples for latent semantic analysis , 2000 .

[5]  Aditya Joshi,et al.  Language Geometry Using Random Indexing , 2016, QI.

[6]  H. P. Luhn.  A new method of recording and searching information , 1953 .

[7]  Jonathan Goldstein,et al.  When Is ''Nearest Neighbor'' Meaningful? , 1999, ICDT.

[8]  T. Landauer,et al.  Indexing by Latent Semantic Analysis , 1990 .

[9]  Geoffrey E. Hinton,et al.  Learning distributed representations of concepts. , 1989 .

[10]  Allan Hanbury,et al.  Exploration of a Threshold for Similarity Based on Uncertainty in Word Embedding , 2017, ECIR.

[11]  Dominic Widdows,et al.  Geometry and Meaning , 2004, Computational Linguistics.

[12]  Gerard Salton,et al.  A vector space model for automatic indexing , 1975, CACM.

[13]  J. M. Kittross.  The measurement of meaning , 1959 .

[14]  Lance J. Rips,et al.  Semantic distance and the verification of semantic relations , 1973 .

[15]  Curt Burgess,et al.  Producing high-dimensional semantic spaces from lexical co-occurrence , 1996 .

[16]  Tommi S. Jaakkola,et al.  Gromov-Wasserstein Alignment of Word Embedding Spaces , 2018, EMNLP.

[17]  Magnus Sahlgren,et al.  The Word-Space Model: using distributional analysis to represent syntagmatic and paradigmatic relations between words in high-dimensional vector spaces , 2006 .

[18]  Anders Søgaard,et al.  On the Limitations of Unsupervised Bilingual Dictionary Induction , 2018, ACL.

[19]  Yoshua Bengio,et al.  A Neural Probabilistic Language Model , 2003, J. Mach. Learn. Res..

[20]  Magnus Sahlgren,et al.  Filaments of Meaning in Word Space , 2008, ECIR.

[21]  Michael N Jones,et al.  Representing word meaning and order information in a composite holographic lexicon. , 2007, Psychological review.

[22]  Tony A. Plate,et al.  Holographic Reduced Representation: Distributed Representation for Cognitive Structures , 2003 .

[23]  David Dubin,et al.  The Most Influential Paper Gerard Salton Never Wrote , 2004, Libr. Trends.