Speaking, Seeing, Understanding: Correlating semantic models with conceptual representation in the brain

Research in computational semantics is increasingly guided by our understanding of human semantic processing. However, semantic models are typically studied in the context of natural language processing system performance. In this paper, we present a systematic evaluation and comparison of a range of widely-used, state-of-the-art semantic models in their ability to predict patterns of conceptual representation in the human brain. Our results provide new insights both for the design of computational semantic models and for further research in cognitive neuroscience.

[1]  Randy Goebel,et al.  Using Visual Information to Predict Lexical Preference , 2011, RANLP.

[2]  Omer Levy,et al.  Dependency-Based Word Embeddings , 2014, ACL.

[3]  Benjamin D. Zinszer,et al.  Representational similarity encoding for fMRI: Pattern-based synthesis to predict brain activity using stimulus-model-similarities , 2016, NeuroImage.

[4]  Carina Silberer,et al.  Grounded Models of Semantic Representation , 2012, EMNLP.

[5]  Massimo Poesio,et al.  Visually Grounded and Textual Semantic Models Differentially Decode Brain Activity Associated with Concrete and Abstract Nouns , 2017, TACL.

[6]  Mark S. Seidenberg,et al.  Semantic feature production norms for a large set of living and nonliving things , 2005, Behavior research methods.

[7]  Anna Korhonen,et al.  Using fMRI activation to conceptual stimuli to evaluate methods for extracting conceptual representations from corpora , 2010, HLT-NAACL 2010.

[8]  Irene Koshik,et al.  Journal of the american society for information science and technology-2012 , 2012 .

[9]  Léon Bottou,et al.  Learning Image Embeddings using Convolutional Neural Networks for Improved Multi-Modal Semantics , 2014, EMNLP.

[10]  Thomas A. Schreiber,et al.  The University of South Florida free association, rhyme, and word fragment norms , 2004, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[11]  Massimo Poesio,et al.  Reading visually embodied meaning from the brain: Visually grounded computational models decode visual-object mental imagery induced by written text , 2015, NeuroImage.

[12]  John A. Carroll,et al.  Applied morphological processing of English , 2001, Natural Language Engineering.

[13]  Richard M. Shiffrin,et al.  Word Association Spaces for Predicting Semantic Similarity Effects in Episodic Memory. , 2005 .

[14]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[15]  Stephen Clark,et al.  Comparing Data Sources and Architectures for Deep Visual Representation Learning in Semantics , 2016, EMNLP.

[16]  Felix Hill,et al.  Learning Abstract Concept Embeddings from Multi-Modal Data: Since You Probably Can’t See What I Mean , 2014, EMNLP.

[17]  Ken McRae,et al.  Category - Specific semantic deficits , 2008 .

[18]  Tom Michael Mitchell,et al.  Predicting Human Brain Activity Associated with the Meanings of Nouns , 2008, Science.

[19]  Lawrence W. Barsalou,et al.  Language and simulation in conceptual processing , 2008 .

[20]  Stephen Clark,et al.  Grounding Semantics in Olfactory Perception , 2015, ACL.

[21]  Massimo Poesio,et al.  Of Words, Eyes and Brains: Correlating Image-Based Distributional Semantic Models with Neural Representations of Concepts , 2013, EMNLP.

[22]  Amy Perfors,et al.  Predicting human similarity judgments with distributional models: The value of word associations. , 2016, COLING.

[23]  Danielle J. Navarro,et al.  Better explanations of lexical and semantic cognition using networks derived from continued rather than single-word associations , 2012, Behavior Research Methods.

[24]  Francisco Pereira,et al.  Using Wikipedia to learn semantic feature representations of concrete concepts in neuroimaging experiments , 2013, Artif. Intell..

[25]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[26]  Stephen Clark,et al.  Vision and Feature Norms: Improving automatic feature norm learning through cross-modal maps , 2016, NAACL.

[27]  T. Landauer,et al.  Indexing by Latent Semantic Analysis , 1990 .

[28]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[29]  Douwe Kiela.  MMFeat: A Toolkit for Extracting Multi-Modal Features , 2016, ACL.

[30]  A. E. Hoerl,et al.  Ridge regression: biased estimation for nonorthogonal problems , 2000 .

[31]  Tom M. Mitchell,et al.  Quantitative modeling of the neural representation of objects: How semantic feature norms can account for fMRI activation , 2011, NeuroImage.

[32]  Mark S. Seidenberg,et al.  Predicting brain activation patterns associated with individual lexical concepts based on five sensory-motor attributes , 2015, Neuropsychologia.

[33]  Gemma Boleda,et al.  Distributional Semantics in Technicolor , 2012, ACL.

[34]  Geoffrey Zweig,et al.  Linguistic Regularities in Continuous Space Word Representations , 2013, NAACL.

[35]  Stephen Clark,et al.  Exploiting Image Generality for Lexical Entailment Detection , 2015, ACL.

[36]  Stephen Clark,et al.  Multi- and Cross-Modal Semantics Beyond Vision: Grounding in Auditory Perception , 2015, EMNLP.

[37]  James R. Curran,et al.  Wide-Coverage Efficient Statistical Parsing with CCG and Log-Linear Models , 2007, Computational Linguistics.

[38]  Geoffrey E. Hinton,et al.  Zero-shot Learning with Semantic Output Codes , 2009, NIPS.

[39]  Jean Maillard,et al.  Black Holes and White Rabbits: Metaphor Identification with Visual Features , 2016, NAACL.

[40]  L. Barsalou.  Grounded cognition. , 2008, Annual review of psychology.

[41]  Tom M. Mitchell,et al.  Selecting Corpus-Semantic Models for Neurolinguistic Decoding , 2012, *SEMEVAL.