Multisensory Statistical Learning: Can Cross-Modal Associations Be Acquired?

Multisensory Statistical Learning: Can Associations between Perceptual Categories Be Acquired? Anne McClure Walk (awalk@slu.edu) Christopher M. Conway (cconway6@slu.edu) Department of Psychology, 3511 Laclede Ave. Saint Louis University Saint Louis, MO 63103 USA Abstract Statistical learning, the process by which people learn patterns of information from their environment that they can apply to new situations, is central to the development of many higher order cognitive skills. Despite a growing research literature, little is still known about how statistical learning operates across perceptual categories. To investigate this issue we assessed college students on their ability to learn a multisensory artificial grammar containing both auditory and visual elements and both within-categorical and cross- categorical associations. The results of Experiment 1 showed that participants were sensitive to grammatically correct test items and ungrammatical test items that contained within- categorical grammatical violations, but were not sensitive to items that contained cross-categorical violations across sensory modalities. Experiment 2 showed that participants were not sensitive to items that contained cross-categorical violations within the same sensory modality. Our findings suggest that multisensory integration across perceptual categories does not occur easily during statistical learning. Keywords: statistical learning, artificial grammar learning, multisensory processing, domain-general Introduction Statistical learning, the ability to detect statistical associations in the environment (Perruchet & Pacton, 2006), appears to be important across a range of cognitive domains, including language, motor skills, and event segmentation (Conway, Pisoni, Anaya, Karpicke, & Henning, 2011; Conway, Bauernschmidt, Huang, & Pisoni, 2010; Leclerq & Majerus, 2010; Zacks & Swallow, 2007). Despite a growing body of research investigating different aspects of statistical learning, little is known about how learning takes place across perceptual categories and sensory modalities. To illustrate the importance of multisensory processing in cognition, we briefly consider its role in speech perception and production, which require the integration of material across perceptual categories. Rosenblum (2008) suggested that spoken language processing is naturally a multisensory phenomenon, pointing out that infants appear to use visual speech cues early in life to help perceive speech. Furthermore, when one sensory modality is insufficient for perceiving a speech element, the other modality can be recruited: for example, phonemes that are auditorily similar tend to be visually distinct in terms of facial and mouth movements. The importance of multisensory processing in speech perception is also seen in the well known McGurk illusion (McGurk, 1976) in which participants see a video of a person’s mouth verbalizing one syllable, while an auditory track is played of a different syllable. When the auditory input does not match the visual input, participants report perceiving a hybrid syllable constructed from combining the visual and auditory information. Clearly, multisensory processing is an important phenomenon. However, it is still unknown to what extent cross-categorical inputs can be integrated in the case of statistical learning. One possibility is that statistical learning is domain general, and therefore operates equally across all modalities and perceptual categories. Under this view, one would expect that multisensory statistical learning would be robust, and that learning would be comparable across domains. Indeed, Seitz, Kim, Wassenhoven, and Shams (2007) used a statistical learning paradigm to demonstrate that participants learned both audio and visual patterns independently when presented with audio-visual pairings, indicating equivalent levels of learning when exposed to stimuli from different sensory modalities. Several studies have also demonstrated improved performance when stimuli are presented in two rather than a single modality (Kim, Seitz, & Shams, 2008; Robinson & Sloutsky, 2007), which could indicate that stimuli in different modalities are integrated together during statistical learning tasks. Furthermore, several studies have shown transfer between sensory domains, suggesting that knowledge resulting from statistical learning processes can be easily integrated across input domains and perceptual categories (Altmann, Dienes, & Good, 1995; Manza & Reber, 1997). On the other hand, recent research suggests that statistical learning may not be purely domain-general. For instance, modality constraints exist which bias and affect how statistical patterns are acquired (Emberson, Conway, & Christiansen, in press; Conway & Christiansen, 2005). The presence of these modality constraints suggest that although learning across perceptual domains might operate using similar computational principles, each modality may also be biased to acquire certain types of information better than others. Even so, whether people are able to learn patterns when cross-categorical dependencies are employed is a less explored issue. Conway and Christiansen (2006) showed that when learning two separate sets of regularities concurrently, participants demonstrated learning only when the two sets of stimuli were in different sensory modalities or perceptual categories. They argued that this demonstrates that statistical learning relies on stimulus-specific rather