Large-scale evaluation of dependency-based DSMs: Are they worth the effort?

This paper presents a large-scale evaluation study of dependency-based distributional semantic models. We evaluate dependency-filtered and dependency-structured DSMs in a number of standard semantic similarity tasks, systematically exploring their parameter space in order to give them a “fair shot” against window-based models. Our results show that properly tuned window-based DSMs still outperform the dependency-based models in most tasks. There appears to be little need for the language-dependent resources and computational cost associated with syntactic analysis.

[1]  Stefan Evert,et al.  Contrasting Syntagmatic and Paradigmatic Relations: Insights from Distributional Semantic Models , 2014, *SEM@COLING.

[2]  Christopher D. Manning,et al.  Generating Typed Dependency Parses from Phrase Structure Parses , 2006, LREC.

[3]  T. Landauer,et al.  A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge. , 1997 .

[4]  Nathan Halko,et al.  Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions , 2009, SIAM Rev..

[5]  Danqi Chen,et al.  A Fast and Accurate Dependency Parser using Neural Networks , 2014, EMNLP.

[6]  Sebastian Padó,et al.  Crosslingual and Multilingual Construction of Syntax-Based Vector Space Models , 2014, TACL.

[7]  Georgiana Dinu,et al.  Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors , 2014, ACL.

[8]  Magnus Sahlgren,et al.  The Word-Space Model: using distributional analysis to represent syntagmatic and paradigmatic relations between words in high-dimensional vector spaces , 2006 .

[9]  Hinrich Schütze,et al.  Unsupervised Classification with Dependency Based Word Spaces , 2009 .

[10]  John A Bullinaria,et al.  Extracting semantic representations from word co-occurrence statistics: stop-lists, stemming, and SVD , 2012, Behavior research methods.

[11]  Joakim Nivre,et al.  An Efficient Algorithm for Projective Dependency Parsing , 2003, IWPT.

[12]  Alessandro Lenci,et al.  Distributional Memory: A General Framework for Corpus-Based Semantics , 2010, CL.

[13]  Stephen Clark,et al.  A Systematic Study of Semantic Vector Space Model Parameters , 2014, CVSC@EACL.

[14]  J. Bullinaria,et al.  Extracting semantic representations from word co-occurrence statistics: A computational study , 2007, Behavior research methods.

[15]  Ehud Rivlin,et al.  Placing search in context: the concept revisited , 2002, TOIS.

[16]  Stefan Evert.  Distributional Semantics in R with the wordspace Package , 2014, COLING.

[17]  Mirella Lapata,et al.  Dependency-Based Construction of Semantic Space Models , 2007, CL.

[18]  Stefan Evert,et al.  Evaluating Neighbor Rank and Distance Measures as Predictors of Semantic Priming , 2013, CMCL.

[19]  Sabine Schulte im Walde,et al.  Exploiting Fine-grained Syntactic Transfer Features to Predict the Compositionality of German Particle Verbs , 2015, IWCS.

[20]  Katrin Erk,et al.  A Flexible, Corpus-Driven Model of Regular and Inverse Selectional Preferences , 2010, CL.

[21]  J. Fox.  Effect Displays in R for Generalised Linear Models , 2003 .

[22]  Ali S. Hadi,et al.  Finding Groups in Data: An Introduction to Chster Analysis , 1991 .

[23]  Stefan Evert,et al.  A Large Scale Evaluation of Distributional Semantic Models: Parameters, Interactions and Model Selection , 2014, TACL.

[24]  Helmut Schmid,et al.  Improvements in Part-of-Speech Tagging with an Application to German , 1999 .

[25]  Ken McRae,et al.  People Use their Knowledge of Common Events to Understand Language, and Do So as Quickly as Possible , 2009, Lang. Linguistics Compass.

[26]  Abdulrahman Almuhareb,et al.  Attributes in lexical acquisition , 2006 .