Symbol Grounding Without Direct Experience: Do Words Inherit Sensorimotor Activation From Purely Linguistic Context?

Theories of embodied cognition assume that concepts are grounded in non-linguistic, sensorimotor experience. In support of this assumption, previous studies have shown that upwards response movements are faster than downwards movements after participants have been presented with words whose referents are typically located in the upper vertical space (and vice versa for downwards responses). This is taken as evidence that processing these words reactivates sensorimotor experiential traces. This congruency effect was also found for novel words, after participants learned these words as labels for novel objects that they encountered either in their upper or lower visual field. While this indicates that direct experience with a word's referent is sufficient to evoke said congruency effects, the present study investigates whether this direct experience is also a necessary condition. To this end, we conducted five experiments in which participants learned novel words from purely linguistic input: Novel words were presented in pairs with real up- or down-words (Experiment 1); they were presented in natural sentences where they replaced these real words (Experiment 2); they were presented as new labels for these real words (Experiment 3); and they were presented as labels for novel combined concepts based on these real words (Experiment 4 and 5). In all five experiments, we did not find any congruency effects elicited by the novel words; however, participants were always able to make correct explicit judgements about the vertical dimension associated to the novel words. These results suggest that direct experience is necessary for reactivating experiential traces, but this reactivation is not a necessary condition for understanding (in the sense of storing and accessing) the corresponding aspects of word meaning.

[1]  Angeliki Lazaridou,et al.  Multimodal Word Meaning Induction From Minimal Exposure to Natural Text. , 2017, Cognitive science.

[2]  Bradford Z. Mahon,et al.  The burden of embodied cognition. , 2015, Canadian journal of experimental psychology = Revue canadienne de psychologie experimentale.

[3]  J. Prinz Furnishing the Mind: Concepts and Their Perceptual Basis , 2004 .

[4]  Jonathan W. Peirce,et al.  PsychoPy—Psychophysics software in Python , 2007, Journal of Neuroscience Methods.

[5]  Alessandro Lenci,et al.  Distributional semantics in linguistic and cognitive research , 2008 .

[6]  B. Kaup,et al.  Root versus roof: automatic activation of location information during word processing , 2011, Psychonomic bulletin & review.

[7]  Guy Dove Three symbol ungrounding problems: Abstract concepts and the future of embodied cognition , 2016, Psychonomic bulletin & review.

[8]  Tom Ziemke,et al.  On the Importance of a Rich Embodiment in the Grounding of Concepts: Perspectives From Embodied Cognitive Science and Computational Linguistics , 2014, Top. Cogn. Sci..

[9]  Barbara Kaup,et al.  Embodiment and second-language: Automatic activation of motor responses during processing spatially associated L2 words and emotion L2 words in a vertical Stroop paradigm , 2014, Brain and Language.

[10]  L. Barsalou,et al.  Whither structured representation? , 1999, Behavioral and Brain Sciences.

[11]  A. Glenberg Few believe the world is flat: How embodiment is changing the scientific understanding of cognition. , 2015, Canadian journal of experimental psychology = Revue canadienne de psychologie experimentale.

[12]  Michael E R Nicholls,et al.  The highs and lows of the interaction between word meaning and space. , 2013, Journal of experimental psychology. Human perception and performance.

[13]  R. Baayen,et al.  Analyzing Reaction Times , 2010 .

[14]  Stevan Harnad The Symbol Grounding Problem , 1999, ArXiv.

[15]  Lawrence W. Barsalou,et al.  Are Automatic Conceptual Cores the Gold Standard of Semantic Processing? The Context-Dependence of Spatial Meaning in Grounded Congruency Effects , 2015, Cogn. Sci..

[16]  Elia Bruni,et al.  Multimodal Distributional Semantics , 2014, J. Artif. Intell. Res..

[17]  T. Landauer,et al.  A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge. , 1997 .

[18]  B. Kaup,et al.  Do I need to have my hands free to understand hand-related language? Investigating the functional relevance of experiential simulations , 2019, Psychological research.

[19]  Zellig S. Harris,et al.  Distributional Structure , 1954 .

[20]  Douglas L. Medin,et al.  3. The Classical View , 1981 .

[21]  Louise Connell,et al.  Flexible and fast: Linguistic shortcut affects both shallow and deep conceptual processing , 2013, Psychonomic bulletin & review.

[22]  Rolf A. Zwaan,et al.  Embodied Language: A Review of the Role of the Motor System in Language Comprehension , 2008, Quarterly journal of experimental psychology.

[23]  Guy Dove Beyond perceptual symbols: A call for representational pluralism , 2009, Cognition.

[24]  G. Murphy,et al.  The Big Book of Concepts , 2002 .

[25]  Michael P. Kaschak,et al.  Grounding language in action , 2002, Psychonomic bulletin & review.

[26]  Lawrence W. Barsalou,et al.  Language and simulation in conceptual processing , 2008 .

[27]  Lawrence W. Barsalou,et al.  Perceptions of perceptual symbols , 1999, Behavioral and Brain Sciences.

[28]  Guy Dove,et al.  Thinking in Words: Language as an Embodied Medium of Thought , 2014, Top. Cogn. Sci..

[29]  W. Kintsch The role of knowledge in discourse comprehension: a construction-integration model. , 1988, Psychological review.

[30]  Carina Silberer,et al.  Grounding the Meaning of Words with Visual Attributes , 2017 .

[31]  Walter Kintsch,et al.  Toward a model of text comprehension and production. , 1978 .

[32]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[33]  Martin H. Fischer,et al.  A hierarchical view of grounded, embodied, and situated numerical cognition , 2012, Cognitive Processing.

[34]  R. Baayen,et al.  Mixed-effects modeling with crossed random effects for subjects and items , 2008 .

[35]  John E. Hummel,et al.  Symbolic Versus Associative Learning , 2010, Cogn. Sci..

[36]  M. Masson Toward a deeper understanding of embodiment. , 2015, Canadian journal of experimental psychology = Revue canadienne de psychologie experimentale.

[37]  Jason Wittenberg,et al.  Clarify: Software for Interpreting and Presenting Statistical Results , 2003 .

[38]  D H Brainard,et al.  The Psychophysics Toolbox. , 1997, Spatial vision.

[39]  Guy Dove On the need for Embodied and Dis-Embodied Cognition , 2011, Front. Psychology.

[40]  Max M. Louwerse,et al.  Symbol Interdependency in Symbolic and Embodied Cognition , 2011, Top. Cogn. Sci..

[41]  Gabriella Vigliocco,et al.  Integrating experiential and distributional data to learn semantic representations. , 2009, Psychological review.

[42]  P. Robert Duimering,et al.  Conceptual combination: Models, theories, and controversies , 2009 .

[43]  Paul Thagard,et al.  Conceptual Combination and Scientific Discovery , 1984, PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association.

[44]  F. Binkofski,et al.  The challenge of abstract concepts. , 2017, Psychological bulletin.

[45]  Thomas K. Landauer,et al.  Word Maturity: A New Metric for Word Knowledge , 2011 .

[46]  Benjamin Naumann The Architecture Of Cognition , 2016 .

[47]  J. Fox Effect Displays in R for Generalised Linear Models , 2003 .

[48]  D. Barr,et al.  Random effects structure for confirmatory hypothesis testing: Keep it maximal. , 2013, Journal of memory and language.

[49]  B. Kaup,et al.  Is It All Task-Specific? The Role of Binary Responses, Verbal Mediation, and Saliency for Eliciting Language-Space Associations , 2017, Journal of experimental psychology. Learning, memory, and cognition.

[50]  R. Rescorla A theory of pavlovian conditioning: The effectiveness of reinforcement and non-reinforcement , 1972 .

[51]  Brendan T. Johns,et al.  Generating structure from experience: A retrieval-based model of language processing. , 2015, Canadian journal of experimental psychology = Revue canadienne de psychologie experimentale.

[52]  Akira Utsumi,et al.  Grounded Distributional Semantics for Abstract Words , 2016, CogSci.

[53]  D. Bates,et al.  Fitting Linear Mixed-Effects Models Using lme4 , 2014, 1406.5823.

[54]  John R. Searle,et al.  Minds, brains, and programs , 1980, Behavioral and Brain Sciences.

[55]  M. Ross Quillian,et al.  Retrieval time from semantic memory , 1969 .

[56]  Anna M. Borghi,et al.  Words as Social Tools: An Embodied View on Abstract Concepts , 2014 .

[57]  Michael N. Jones,et al.  Perceptual Inference Through Global Lexical Similarity , 2012, Top. Cogn. Sci..

[58]  G. Lakoff,et al.  Metaphors We Live by , 1981 .

[59]  B. Kaup,et al.  Forming associations between language and sensorimotor traces during novel word learning , 2016, Language and Cognition.

[60]  J. Stroop Studies of interference in serial verbal reactions. , 1992 .

[61]  Mark Steyvers,et al.  Topics in semantic representation. , 2007, Psychological review.

[62]  Rolf A. Zwaan,et al.  Embodied Sentence Comprehension , 2005 .

[63]  Magnus Sahlgren,et al.  The Distributional Hypothesis , 2008 .

[64]  Marc Brysbaert,et al.  Wuggy: A multilingual pseudoword generator , 2010, Behavior research methods.

[65]  R. Sternberg Most vocabulary is learned from context. , 1987 .

[66]  J. Hampton Conceptual combination , 2003 .

[67]  Mark S. Seidenberg,et al.  Semantic feature production norms for a large set of living and nonliving things , 2005, Behavior research methods.

[68]  E. Capaldi,et al.  The organization of behavior. , 1992, Journal of applied behavior analysis.

[69]  Curt Burgess,et al.  Producing high-dimensional semantic spaces from lexical co-occurrence , 1996 .

[70]  P. Johnson-Laird,et al.  Mental Models: Towards a Cognitive Science of Language, Inference, and Consciousness , 1985 .

[71]  A. Glenberg,et al.  Symbol Grounding and Meaning: A Comparison of High-Dimensional and Embodied Theories of Meaning , 2000 .

[72]  Xu Xu,et al.  Content Differences for Abstract and Concrete Concept , 2005, Cogn. Sci..

[73]  M R Quillian,et al.  Word concepts: a theory and simulation of some basic semantic capabilities. , 1967, Behavioral science.

[74]  Patricia A. Herman,et al.  Learning Words from Context. , 1985 .

[75]  Lera Boroditsky,et al.  Semantic Coherence Facilitates Distributional Learning. , 2017, Cognitive science.