Modality Switch Effects Emerge Early and Increase throughout Conceptual Processing: Evidence from ERPs

We tested whether conceptual processing is modality-specific by tracking the time course of the Conceptual Modality Switch effect. Forty-six participants verified the relation between property words and concept words. The conceptual modality of consecutive trials was manipulated in order to produce an Auditory-to-visual switch condition, a Haptic-to-visual switch condition, and a Visual-to-visual, no-switch condition. Event-Related Potentials (ERPs) were time-locked to the onset of the first word (property) in the target trials so as to measure the effect online and to avoid a within-trial confound. A switch effect was found, characterized by more negative ERP amplitudes for modality switches than no-switches. It proved significant in four typical time windows from 160 to 750 milliseconds post word onset, with greater strength in posterior brain regions, and after 350 milliseconds. These results suggest that conceptual processing may be modality-specific in certain tasks, but also that the early stage of processing is relatively amodal.

[1]  Nikola Vukovic,et al.  Primary motor cortex functionally contributes to language comprehension: An online rTMS study , 2017, Neuropsychologia.

[2]  L. Barsalou,et al.  Verifying Different-Modality Properties for Concepts Produces Switching Costs , 2003, Psychological science.

[3]  Max M. Louwerse,et al.  A Taste of Words: Linguistic Context and Perceptual Simulation Predict the Modality of Words , 2011, Cogn. Sci..

[4]  Alan Garnham,et al.  Switching Modalities in A Sentence Verification Task: ERP Evidence for Embodied Language Processing , 2011, Front. Psychology.

[5]  L. Barsalou.  On Staying Grounded and Avoiding Quixotic Dead Ends , 2016, Psychonomic Bulletin & Review.

[6]  O. Hauk.  Only time will tell – why temporal information is essential for our neuroscientific understanding of semantics , 2016, Psychonomic Bulletin & Review.

[7]  Ludovica Labruna,et al.  A Functional Role for the Motor System in Language Understanding , 2011, Psychological science.

[8]  R. Nicoletti,et al.  The modality-switch effect: visually and aurally presented prime sentences activate our senses , 2015, Front. Psychol..

[9]  Harold Bekkering,et al.  Flexibility in Embodied Language Processing: Context Effects in Lexical Access , 2014, Top. Cogn. Sci..

[10]  Dermot Lynott,et al.  Modality exclusivity norms for 423 object properties , 2009, Behavior research methods.

[11]  Marta Kutas,et al.  Empirically grounding grounded cognition: The case of color , 2014, NeuroImage.

[12]  L. Connell,et al.  Modality exclusivity norms for 400 nouns: The relationship between perceptual experience and surface word form , 2012, Behavior Research Methods.

[13]  Bradford Z. Mahon,et al.  A critical look at the embodied cognition hypothesis and a new proposal for grounding conceptual content , 2008, Journal of Physiology-Paris.

[14]  Max M. Louwerse,et al.  Verifying properties from different emotions produces switching costs: Evidence for coarse-grained language statistics and fine-grained perceptual simulation , 2013, CogSci.

[15]  A. Garnham,et al.  Exploring Modality Switching Effects in Negated Sentences: Further Evidence for Grounded Representations , 2013, Front. Psychol..

[16]  Seana Coulson,et al.  Modality Switching in a Property Verification Task: An ERP Study of What Happens When Candles Flicker after High Heels Click , 2011, Front. Psychology.

[17]  M. Louwerse,et al.  Neurological Evidence Linguistic Processes Precede Perceptual Simulation in Conceptual Processing , 2012, Front. Psychology.

[18]  R. Nicoletti,et al.  The Multilevel Modality-Switch Effect: What Happens When We See the Bees Buzzing and Hear the Diamonds Glistening , 2017, Psychonomic bulletin & review.