Modality Switching Costs Emerge in Concept Creation as Well as Retrieval

Theories of embodied cognition hold that the conceptual system uses perceptual simulations for the purposes of representation. A strong prediction is that perceptual phenomena should emerge in conceptual processing, and, in support, previous research has shown that switching modalities from one trial to the next incurs a processing cost during conceptual tasks. However, to date, such research has been limited by its reliance on the retrieval of familiar concepts. We therefore examined concept creation by asking participants to interpret modality-specific compound phrases (i.e., conceptual combinations). Results show that modality switching costs emerge during the creation of new conceptual entities: People are slower to simulate a novel concept (e.g., auditory jingling onion) when their attention has already been engaged by a different modality in simulating a familiar concept (e.g., visual shiny penny). Furthermore, these costs cannot be accounted for by linguistic factors alone. Rather, our findings support the embodied view that concept creation, as well as retrieval, requires situated perceptual simulation.

[1]  Rolf A. Zwaan,et al.  Grounding Cognition: Introduction to Grounding Cognition: The Role of Perception and Action in Memory, Language, and Thinking , 2005 .

[2]  C. Spence,et al.  The cost of expecting events in the wrong sensory modality , 2001, Perception & psychophysics.

[3]  L. Connell,et al.  Is a bear white in the woods? Parallel representation of implied object color during language comprehension , 2009, Psychonomic bulletin & review.

[4]  Louise Connell,et al.  Look but don’t touch: Tactile disadvantage in processing modality-specific words , 2010, Cognition.

[5]  Charles Wallis,et al.  Computation and cognition , 2003, J. Exp. Theor. Artif. Intell..

[6]  I. Johnsrude,et al.  Somatotopic Representation of Action Words in Human Motor and Premotor Cortex , 2004, Neuron.

[7]  Friedemann Pulvermüller,et al.  Reading cinnamon activates olfactory brain regions , 2006, NeuroImage.

[8]  L. Barsalou,et al.  Verifying Different-Modality Properties for Concepts Produces Switching Costs , 2003, Psychological science.

[9]  Lawrence W. Barsalou,et al.  Perceptual Processing Affects Conceptual Processing , 2008, Cogn. Sci..

[10]  Dermot Lynott,et al.  Embodied Conceptual Combination , 2010, Front. Psychology.

[11]  James A. Bovaird,et al.  On the use of multilevel modeling as an alternative to items analysis in psycholinguistic research , 2007, Behavior research methods.

[12]  Thomas A. Schreiber,et al.  The University of South Florida free association, rhyme, and word fragment norms , 2004, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[13]  E. Wisniewski When concepts combine , 1997, Psychonomic bulletin & review.

[14]  J. Fodor,et al.  The structure of a semantic theory , 1963 .

[15]  J. F. Marques,et al.  Specialization and semantic organization: Evidence for multiple semantics linked to sensory modalities , 2006, Memory & cognition.

[16]  Michael P. Kaschak,et al.  Grounding language in action , 2002, Psychonomic bulletin & review.

[17]  Michael P. Kaschak,et al.  Perception of motion affects language processing , 2005, Cognition.

[18]  G. Lakoff,et al.  The Brain's concepts: the role of the Sensory-motor system in conceptual knowledge , 2005, Cognitive neuropsychology.

[19]  Walter Kintsch,et al.  Toward a model of text comprehension and production. , 1978 .

[20]  Allen Newell,et al.  Human Problem Solving. , 1973 .

[21]  R. Baayen,et al.  Mixed-effects modeling with crossed random effects for subjects and items , 2008 .

[22]  R. Gibbs,et al.  Embodied experience and linguistic meaning , 2003, Brain and Language.

[23]  L. Barsalou,et al.  Perceptual simulation in conceptual combination: evidence from property generation. , 2009, Acta psychologica.

[24]  Bruce Bridgeman,et al.  Space-independent modality-driven attentional capture in auditory, tactile and visual systems , 2004, Experimental Brain Research.

[25]  L. Connell Representing object colour in language comprehension , 2007, Cognition.

[26]  W. Schneider,et al.  Distinct and common cortical activations for multimodal semantic categories , 2006, Cognitive, affective & behavioral neuroscience.

[27]  Lawrence W. Barsalou,et al.  Language and simulation in conceptual processing , 2008 .

[28]  Diane Pecher,et al.  Verifying Properties from Different Modalities for Concepts Produces Switching Costs , 2002 .

[29]  G. Miller,et al.  Cognitive science. , 1981, Science.

[30]  J. Fodor,et al.  The Language of Thought , 1980 .

[31]  Rolf A. Zwaan,et al.  PSYCHOLOGICAL SCIENCE Research Article THE EFFECT OF IMPLIED ORIENTATION DERIVED FROM VERBAL CONTEXT ON PICTURE RECOGNITION , 2022 .

[32]  M. Ross Quillian,et al.  Retrieval time from semantic memory , 1969 .

[33]  E. Tulving,et al.  Episodic and semantic memory , 1972 .

[34]  W. K. Simmons,et al.  A common neural substrate for perceiving and knowing about color , 2007, Neuropsychologia.

[35]  Mark T. Keane,et al.  Familiarity and relational preference in the understanding of noun-noun compounds , 2006, Memory & cognition.

[36]  Mark T. Keane,et al.  Efficient creativity: constraint-guided conceptual combination , 2000, Cogn. Sci..

[37]  Dermot Lynott,et al.  Modality exclusivity norms for 423 object properties , 2009, Behavior research methods.

[38]  Lance J. Rips,et al.  Combining Prototypes: A Selective Modification Model , 1988, Cogn. Sci..

[39]  Patrick Jeuniaux,et al.  Language comprehension is both embodied and symbolic , 2008 .

[40]  Dermot Lynott,et al.  The Effect of Prosody on Conceptual Combination , 2010, Cogn. Sci..

[41]  L. Barsalou,et al.  Whither structured representation? , 1999, Behavioral and Brain Sciences.

[42]  Nicolas Vermeulen,et al.  Switching Between Sensory and Affective Systems Incurs Processing Costs , 2007, Cogn. Sci..

[43]  R. Gibbs Embodiment and Cognitive Science: Concepts , 2005 .

[44]  Max M. Louwerse,et al.  A Taste of Words: Linguistic Context and Perceptual Simulation Predict the Modality of Words , 2011, Cogn. Sci..