Distributional semantics from text and images

We present a distributional semantic model combining text- and image-based features. We evaluate this multimodal semantic model on simulating similarity judgments, concept clustering and the BLESS benchmark. When integrated with the same core text-based model, image-based features are at least as good as further text-based features, and they capture different qualitative aspects of the tasks, suggesting that the two sources of information are complementary.

[1]  H. J. Arnold.  Introduction to the Practice of Statistics , 1990 .

[2]  G. Miller,et al.  Contextual correlates of semantic similarity , 1991 .

[3]  T. Landauer,et al.  A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge. , 1997 .

[4]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[5]  G. Karypis,et al.  Criterion Functions for Document Clustering ∗ Experiments and Analysis , 2001 .

[6]  George Karypis,et al.  CLUTO - A Clustering Toolkit , 2002 .

[7]  Ehud Rivlin,et al.  Placing search in context: the concept revisited , 2002, TOIS.

[8]  Rolf A. Zwaan.  The Immersed Experiencer: Toward An Embodied Theory Of Language Comprehension , 2003 .

[9]  Andrew Zisserman,et al.  Video Google: a text retrieval approach to object matching in videos , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[10]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[11]  Gabriela Csurka,et al.  Visual categorization with bags of keypoints , 2002, eccv 2004.

[12]  Laura A. Dabbish,et al.  Labeling images with a computer game , 2004, AAAI Spring Symposium: Knowledge Collection from Volunteer Contributors.

[13]  Thomas Hofmann,et al.  Unsupervised Learning by Probabilistic Latent Semantic Analysis , 2004, Machine Learning.

[14]  G. Karypis,et al.  Criterion functions for document clustering , 2005 .

[15]  Trevor Darrell,et al.  The pyramid match kernel: discriminative classification with sets of image features , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[16]  Abdulrahman Almuhareb,et al.  Attributes in lexical acquisition , 2006 .

[17]  David Nistér,et al.  Scalable Recognition with a Vocabulary Tree , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[18]  S. Harnad.  Symbol grounding problem , 1991, Scholarpedia.

[19]  Chong-Wah Ngo,et al.  Evaluating bag-of-visual-words representations in scene classification , 2007, MIR '07.

[20]  Mark Steyvers,et al.  Topics in semantic representation. , 2007, Psychological review.

[21]  Andrew Zisserman,et al.  Image Classification using Random Forests and Ferns , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[22]  Alessandro Lenci,et al.  Concepts and properties in word spaces , 2008 .

[23]  Lawrence W. Barsalou,et al.  Language and simulation in conceptual processing , 2008 .

[24]  Jonathon S. Hare,et al.  Semantic spaces revisited: investigating the performance of auto-annotation and semantic retrieval using semantic spaces , 2008, CIVR '08.

[25]  Gabriella Vigliocco,et al.  Integrating experiential and distributional data to learn semantic representations. , 2009, Psychological review.

[26]  Eneko Agirre,et al.  A Study on Similarity and Relatedness Using Distributional and WordNet-based Approaches , 2009, NAACL.

[27]  Yansong Feng,et al.  Visual Information in Semantic Representation , 2010, NAACL.

[28]  Patrick Pantel,et al.  From Frequency to Meaning: Vector Space Models of Semantics , 2010, J. Artif. Intell. Res..

[29]  Alessandro Lenci,et al.  Distributional Memory: A General Framework for Corpus-Based Semantics , 2010, CL.

[30]  Massimo Poesio,et al.  Strudel: A distributional semantic model based on properties and types , 2010 .

[31]  Andrea Vedaldi,et al.  Vlfeat: an open and portable library of computer vision algorithms , 2010, ACM Multimedia.

[32]  Max M. Louwerse,et al.  Symbol Interdependency in Symbolic and Embodied Cognition , 2011, Top. Cogn. Sci..

[33]  Richard Szeliski,et al.  Computer Vision - Algorithms and Applications , 2011, Texts in Computer Science.

[34]  M. Engelmann.  The Philosophical Investigations , 2013 .