Modality Switching in a Property Verification Task: An ERP Study of What Happens When Candles Flicker after High Heels Click

The perceptual modalities associated with property words, such as flicker or click, have previously been demonstrated to affect subsequent property verification judgments (Pecher et al., 2003). Known as the conceptual modality switch effect, this finding supports the claim that brain systems for perception and action help subserve the representation of concepts. The present study addressed the cognitive and neural substrate of this effect by recording event-related potentials (ERPs) as participants performed a property verification task with visual or auditory properties in key trials. We found that for visual property verifications, modality switching was associated with an increased amplitude N400. For auditory verifications, switching led to a larger late positive complex. Observed ERP effects of modality switching suggest property words access perceptual brain systems. Moreover, the timing and pattern of the effects suggest perceptual systems impact the decision-making stage in the verification of auditory properties, and the semantic stage in the verification of visual properties.

[1]  P. Holcomb.  Automatic and attentional processing: An event-related brain potential analysis of semantic priming , 1988, Brain and Language.

[2]  L. Barsalou,et al.  Verifying Different-Modality Properties for Concepts Produces Switching Costs , 2003, Psychological science.

[3]  C. Spence,et al.  The cost of expecting events in the wrong sensory modality , 2001, Perception & psychophysics.

[4]  Marta Kutas,et al.  The intractability of scaling scalp distributions to infer neuroelectric sources. , 2002, Psychophysiology.

[5]  Mark W. Greenlee,et al.  Modality shift effects mimic multisensory interactions: an event-related potential study , 2007, Experimental Brain Research.

[6]  G. Murphy,et al.  The Big Book of Concepts , 2002 .

[7]  L. Barsalou.  Grounded cognition. , 2008, Annual review of psychology.

[8]  Kara D. Federmeier,et al.  Electrophysiology reveals semantic memory use in language comprehension , 2000, Trends in Cognitive Sciences.

[9]  J. Polich,et al.  P3a and P3b from typical auditory and visual stimuli , 1999, Clinical Neurophysiology.

[10]  Kara D. Federmeier,et al.  Thirty years and counting: finding meaning in the N400 component of the event-related brain potential (ERP). , 2011, Annual review of psychology.

[11]  Diane Pecher,et al.  Sensorimotor simulations underlie conceptual representations: Modality-specific effects of prior activation , 2004, Psychonomic bulletin & review.

[12]  P. Hagoort.  The fractionation of spoken language understanding by measuring electrical and magnetic brain signals , 2008, Philosophical Transactions of the Royal Society B: Biological Sciences.

[13]  Kara D. Federmeier,et al.  A Rose by Any Other Name: Long-Term Memory Structure and Sentence Processing , 1999 .

[14]  Ying Choon Wu,et al.  Meaningful gestures: electrophysiological indices of iconic gesture comprehension. , 2005, Psychophysiology.

[15]  M. Kutas,et al.  Brain potentials during reading reflect word expectancy and semantic association , 1984, Nature.

[16]  Dermot Lynott,et al.  Modality exclusivity norms for 423 object properties , 2009, Behavior research methods.

[17]  Brigitte Röder,et al.  The redundant target effect is affected by modality switch costs , 2004, Psychonomic bulletin & review.

[18]  Irene P. Kan,et al.  ROLE OF MENTAL IMAGERY IN A PROPERTY VERIFICATION TASK: FMRI EVIDENCE FOR PERCEPTUAL REPRESENTATIONS OF CONCEPTUAL KNOWLEDGE , 2003, Cognitive neuropsychology.

[19]  J. Polich.  Updating P300: An integrative theory of P3a and P3b , 2007, Clinical Neurophysiology.

[20]  B. Mesquita,et al.  Adjustment to Chronic Diseases and Terminal Illness Health Psychology : Psychological Adjustment to Chronic Disease , 2006 .

[21]  Diane Pecher,et al.  Verifying Properties from Different Modalities for Concepts Produces Switching Costs , 2002 .

[22]  P. Rodway.  The modality shift effect and the effectiveness of warning signals in different modalities. , 2005, Acta psychologica.

[23]  T. Gunter,et al.  The communicative style of a speaker can affect language comprehension? ERP evidence from the comprehension of irony , 2010, Brain Research.

[24]  Shlomo Bentin,et al.  Event-related potentials, semantic processes, and expectancy factors in word recognition , 1987, Brain and Language.

[25]  Ellen F. Lau,et al.  A cortical network for semantics: (de)constructing the N400 , 2008, Nature Reviews Neuroscience.

[26]  W. K. Simmons,et al.  A common neural substrate for perceiving and knowing about color , 2007, Neuropsychologia.

[27]  Lawrence W. Barsalou,et al.  Perceptual Processing Affects Conceptual Processing , 2008, Cogn. Sci..

[28]  J. Polich.  Updating P 300 : An Integrative Theory of P 3 a and P 3 b , 2009 .

[29]  L. Barsalou,et al.  Whither structured representation? , 1999, Behavioral and Brain Sciences.

[30]  Nicolas Vermeulen,et al.  Switching Between Sensory and Affective Systems Incurs Processing Costs , 2007, Cogn. Sci..

[31]  H. Neville,et al.  Language and , 2019, Adventure Diffusion.

[32]  A. A. Wijers,et al.  Visual semantic features are activated during the processing of concrete words: event-related potential evidence for perceptual semantic priming. , 2000, Brain research. Cognitive brain research.

[33]  Christine D. Wilson,et al.  Grounding conceptual knowledge in modality-specific systems , 2003, Trends in Cognitive Sciences.

[34]  W. Schneider,et al.  Perceptual Knowledge Retrieval Activates Sensory Brain Regions , 2006, The Journal of Neuroscience.

[35]  Diane Pecher,et al.  Concepts are not represented by conscious imagery , 2009, Psychonomic bulletin & review.

[36]  E Halgren,et al.  Dissociation of recognition memory components following temporal lobe lesions. , 1989, Journal of experimental psychology. Learning, memory, and cognition.