Verifying Properties from Different Modalities for Concepts Produces Switching Costs

According to perceptual symbol systems (Barsalou, 1999), sensory-motor simulations underlie the representation of concepts. It follows that sensory-motor phenomena should arise in conceptual processing. Previous studies have shown that switching from one modality to another during perceptual processing incurs a processing cost. If perceptual simulation underlies conceptual processing, then verifying the properties of concepts should exhibit a switching cost as well. For example, verifying a property in the auditory modality (e.g., BLENDER-loud) should be slower after verifying a property in a different modality (e.g., CRANBERRIES-tart) than in the same modality (e.g., LEAVES-rustling). Only words were presented to subjects, and there were no instructions to use imagery. Nevertheless switching modalities incurred a cost, analogous to switching modalities in perception. A second experiment showed that this effect was not due to associative priming between properties in the same modality. These results support the hypothesis that perceptual simulation underlies conceptual processing. Switching costs in property verification 3 Modern psychology relies heavily on the digital computer as a metaphor for human cognition (e.g., Fodor, 1975; Pylyshyn, 1984). According to this view, the software of the mind can be distinguished from the hardware of the body, with mental representations being amodal redescriptions of sensory-motor experience. Increasingly, however, researchers argue that this approach is fundamentally wrong, suggesting instead that interactions between sensory-motor systems and the physical world underlie cognition. For example, Barsalou’s (1999) theory of perceptual symbol systems proposes that conceptual knowledge is grounded in sensory-motor systems. To represent a concept, neural systems partially run as if interacting with an actual instance. For example, to represent the concept CHAIR, neural systems for vision, action, touch, and emotion partially reenact the experience of a chair. Increasingly, behavioral evidence supports this view (e.g., Klatzky, Pellegrino, McKloskey, & Doherty, 1989; Solomon & Barsalou, 2001, 2002; Spivey, Tyler, Richardson, & Young, 2000; Stanfield & Zwaan, 2001; Wu & Barsalou, 2002; Zwaan, Stanfield, & Yaxleu, in press), as does neural evidence (e.g., Martin, 2001; Martin & Chao, 2001; Martin, Ungerleider, & Haxby, 2000; Pulvermüller, 1999). See Barsalou (1999; in press) and Glenberg (1997) for further evidence. Several aspects of sensory-motor simulations are important for the experiments presented shortly. First, simulations are componential, not holistic. Rather than being like a holistic video recording, a simulation contains many small elements of perception— perceptual symbols—organized coherently. Second, perceptual symbols arise on all modalities of experience—vision, audition, smell, taste, touch, action, emotion, introspection, etc. Third, perceptual symbols vary in accessibility. On a given occasion, only those perceptual symbols most active enter a simulation, such that the simulations of a concept vary considerably across occasions. Furthermore—and most importantly for our purposes—the modalities represented in simulations vary as well. On one occasion, the simulation of a concept might focus on how an object looks (e.g., a LEMON is yellow); on another occasion, a simulation might focus on how the object tastes (e.g., a LEMON is sour). Although multiple modalities may typically be represented, one may often be more salient than others. Switching costs in property verification 4 Furthermore, over time, the focus may remain in a single modality, or it may switch from one modality to another. If switching between modalities occurs during conceptual processing, then a phenomenon from the perception literature is relevant. Spence, Nicholls, and Driver (2000) had subjects discriminate whether a signal occurred on the left or the right in any of three modalities monitored simultaneously (i.e., a light in vision, a touch on a finger, a tone in audition). When two consecutive signals occurred on the same modality, processing stayed within a single system. When consecutive signals occurred on different modalities, processing had to switch between systems. Most importantly, Spence et al. found that switching modalities incurred a cost: Detecting a signal was slower when the previous signal was on a different modality than on the same (also see Spence & Driver, 1998). If conceptual processing utilizes sensory-motor systems, then an analogous cost should occur when conceptual processing switches from one modality to another. To investigate this prediction, we used the property verification task. On target trials, subjects verified a property in one of six modalities (vision, audition, taste, smell, touch, action). For example, subjects might verify the auditory property loud for BLENDER. On the previous trial, subjects either verified a property from a different concept on the same modality or on a different modality (e.g., LEAVES-rustling versus CRANBERRIES-tart). Table 1 provides examples of the critical materials. Because the concepts on the two trials were always unassociated, no associative priming between concepts should occur. Also a high ratio of filler trials to critical trials masked the purpose of the experiment (i.e., the number of paired trials on the same modality was relatively small). The key prediction was that having to switch modalities would slow verification time, relative to staying within the same modality, analogous to modality-switching costs in perceptual processing. Experiment 1 also explored whether the stimulus onset asynchrony (SOA) between presentation of the concept and presentation of the property is a factor in switching costs. Perhaps switching costs disappear when properties lag behind concepts, because the concept has longer to activate properties across modalities. Alternatively, switching costs may remain constant across SOAs if subjects do not commit to a dominant modality until receiving the Switching costs in property verification 5 property word. To assess these possibilities, some subjects received the concept and property on each trial simultaneously (SOA=0 ms), whereas others received the concept first, followed by the property 260 ms later (SOA=260 ms). ----------------------------------------------Insert Table 1 about here ----------------------------------------------Experiment 1 Method Subjects and design. Sixty-four volunteers from Emory University participated for course credit. Thirty-two were assigned randomly to each of the two between-subjects conditions for SOA. Same versus different modality was manipulated within subjects, with equal numbers receiving each counterbalanced version of the list. Materials. A set of 100 concept-property items was developed. Each property was more salient on one modality than on the others. We selected 26 properties from vision, 24 from motor actions, 18 from audition, 12 from touch, 12 from taste, and 8 from smell. Because some modalities have more words for properties than others, the number of properties differed across modalities by necessity. From the 100 concept-property items, 50 pairs were formed. Half contained two properties from the same modality; half contained properties from different modalities. The two items forming a same-modality pair were chosen randomly from items on the relevant modality. According to the norms of Nelson, McEvoy, and Schreiber (1999), the properties in these pairs were not associated. One item in each same-modality pair was randomly assigned to be presented first (the context item), and the other to be presented second (the target item). Table 1 presents an example from each modality. The two items comprising a differentmodality pair were chosen randomly from the remaining items. In pairs of both types, if the two concepts exhibited a relation, they were replaced with items having no relation. Two lists were created such that each target had a same modality context in one list but a differentmodality context in the other. Thus each target item appeared with both same-modality and different-modality contexts, counterbalanced across lists. All critical properties were true of their respective concepts. Switching costs in property verification 6 The experimental trials included 150 pairs, with 50 being critical, for a total of 300 trials. The remaining 100 pairs were fillers, designed to mask the nature of the experiment. Within the filler pairs, 50 contained two false items, 25 contained a true item then a false item, and 25 contained a false item then a true item. Thus true and false responses were equally likely overall. Properties in the fillers sometimes referred to a specific modality but also referred to properties that are represented on multiple modalities (e.g., CAMERAcompact, TOY-plastic, MAP-complicated). To ensure that subjects actually verified the properties of concepts (Solomon & Barsalou, 2002), the concept and property in many false items were related (e.g., OVEN-baked, BUFFALO-winged, BUTTERFLY-bird). The critical and filler pairs were randomly intermixed for each subject. All concepts and properties were used only once. The practice trials consisted of 24 true items and 24 false items, similar in nature to the experimental trials. Procedure. Each trial began with a fixation stimulus (* * * * *) two lines above where the concept name would appear. After 500 ms, the fixation stimulus disappeared. In the 0 ms SOA condition, three lines of text appeared aligned vertically, each two lines apart. The first line contained the concept word in upper case; the second line contained the words “can be” in lowercase; the third line contained the property word in upper case. In the 260 ms SOA condition, the concept word appeared for 160 ms, then “can be” was added for 100 ms, then the property name was added. RTs in all cases were measured from th