Brain Electric Microstate and Perception of Simultaneously Audiovisual Presentation

Associations between picture and sound form the basis of reading. Learning the correspondences between them is a crucial step in reading acquisition. This study was designed to investigate whether task-related processing of audio and visual features was independent or task-related processing in one modality might influence the processing of the other. The present study employed simultaneous audio-visual stimulus in the oddball paradigm to re-examine the effects of attention on audio, visual and audio-visual perception in the non-musician brain. Electroencephalographic (EEG) was recorded from 28 normal participants. None of them had more than three years of formal musical training and none had any musical training within the past five years. Chinese and Korean subjects were presented with tones (auditory: A), pictures (visual: V), and simultaneous tones and pictures (audio-visual: AV). The neural basis of this interaction was investigated by subtracting the event-related potentials (ERPs) to the A and the V stimuli alone from the ERP to the combined AV stimuli (i.e. interaction = AV - (A+V)). The Korean group showed larger mean interaction amplitude and longer in time than the Chinese group. This reveals that experience influences the early cortical automatic processing of linguistically relevant suprasegmental pitch contour. These results suggest that efficient processing of associations between pictures and sounds relies on neural mechanisms similar to those naturally evolved for integrating audiovisual perception.

[1]  J. Polich.  Detection of change : event-related potential and fMRI findings , 2003 .

[2]  R. Hari,et al.  The Human Auditory Sensory Memory Trace Persists about 10 sec: Neuromagnetic Evidence , 1993, Journal of Cognitive Neuroscience.

[3]  G Nyman,et al.  Mismatch negativity (MMN) for sequences of auditory and visual stimuli: evidence for a mechanism specific to the auditory modality. , 1990, Electroencephalography and clinical neurophysiology.

[4]  D. Lehmann,et al.  Adaptive segmentation of spontaneous EEG map series into spatially defined microstates. , 1993, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[5]  István Czigler,et al.  Memory-based detection of task-irrelevant visual changes. , 2002, Psychophysiology.

[6]  R. Ilmoniemi,et al.  Processing of complex sounds in the human auditory cortex as revealed by magnetic brain responses. , 1996, Psychophysiology.

[7]  F. Denoth,et al.  Cerebral psychophysiology: studies in event-related potentials. , 1986, Electroencephalography and clinical neurophysiology. Supplement.

[8]  R. Näätänen.  The role of attention in auditory information processing as revealed by event-related potentials and other brain measures of cognitive function , 1990, Behavioral and Brain Sciences.

[9]  G. Calvert,et al.  Multisensory integration: methodological approaches and emerging principles in the human brain , 2004, Journal of Physiology-Paris.

[10]  D. Javitt,et al.  The Mismatch Negativity of Event‐Related Potentials as a Probe of Transient Auditory Memory: A Review , 1995, Ear and hearing.

[11]  R. Hari,et al.  Seeing speech: visual information from lip movements modifies activity in the human auditory cortex , 1991, Neuroscience Letters.

[12]  A. Fort,et al.  Bimodal speech: early suppressive visual effects in human auditory cortex , 2004, The European journal of neuroscience.

[13]  M. Scherg,et al.  A Source Analysis of the Late Human Auditory Evoked Potentials , 1989, Journal of Cognitive Neuroscience.

[14]  Risto Näätänen,et al.  Simultaneous storage of two complex temporal sound patterns in auditory sensory memory , 2002, Neuroreport.

[15]  M. Sams,et al.  Time course of multisensory interactions during audiovisual speech perception in humans: a magnetoencephalographic study , 2004, Neuroscience Letters.

[16]  Klucharev Vasily,et al.  Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception. , 2003 .

[17]  Gregory McCarthy,et al.  Graded Visual Attention Modulates Brain Responses Evoked by Task-irrelevant Auditory Pitch Changes , 2005, Journal of Cognitive Neuroscience.

[18]  David Poeppel,et al.  Visual speech speeds up the neural processing of auditory speech. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[19]  M. Tervaniemi,et al.  Preattentive periodicity detection in auditory patterns as governed by time and intensity information. , 1996, Brain research. Cognitive brain research.

[20]  E. Amenedo,et al.  MMN in the visual modality: a review , 2003, Biological Psychology.

[21]  John J. Foxe,et al.  Multisensory auditory-somatosensory interactions in early cortical processing revealed by high-density electrical mapping. , 2000, Brain research. Cognitive brain research.

[22]  K. Alho,et al.  Intermodal selective attention. I. Effects on event-related potentials to lateralized auditory and visual stimuli. , 1992, Electroencephalography and clinical neurophysiology.

[23]  A Tales,et al.  Mismatch negativity in the visual modality. , 1999, Neuroreport.

[24]  R. Näätänen.  Attention and brain function , 1992 .

[25]  R. Näätänen,et al.  Intermodal selective attention. II. Effects of attentional load on processing of auditory and visual stimuli in central space. , 1992, Electroencephalography and clinical neurophysiology.

[26]  István Czigler,et al.  Visual change detection: event-related potentials are dependent on stimulus location in humans , 2004, Neuroscience Letters.

[27]  Dirk J. Heslenfeld,et al.  Visual Mismatch Negativity. , 2003 .

[28]  C Alain,et al.  Brain indices of automatic pattern processing , 1994, Neuroreport.

[29]  H. Heinze,et al.  An event-related brain potential study of visual selective attention to conjunctions of color and shape. , 1999, Psychophysiology.

[30]  M. Giard,et al.  Auditory-Visual Integration during Multimodal Object Recognition in Humans: A Behavioral and Electrophysiological Study , 1999, Journal of Cognitive Neuroscience.

[31]  R. Ilmoniemi,et al.  Superior formation of cortical memory traces for melodic patterns in musicians. , 2001, Learning & memory.

[32]  R. Cammann.  Is there a mismatch negativity (MMN) in visual modality? , 1990, Behavioral and Brain Sciences.