Predictive coding of visual–auditory and motor-auditory events: An electrophysiological study

The amplitude of auditory components of the event-related potential (ERP) is attenuated when sounds are self-generated compared to externally generated sounds. This effect has been ascribed to internal forward modals predicting the sensory consequences of one's own motor actions. Auditory potentials are also attenuated when a sound is accompanied by a video of anticipatory visual motion that reliably predicts the sound. Here, we investigated whether the neural underpinnings of prediction of upcoming auditory stimuli are similar for motor-auditory (MA) and visual-auditory (VA) events using a stimulus omission paradigm. In the MA condition, a finger tap triggered the sound of a handclap whereas in the VA condition the same sound was accompanied by a video showing the handclap. In both conditions, the auditory stimulus was omitted in either 50% or 12% of the trials. These auditory omissions induced early and mid-latency ERP components (oN1 and oN2, presumably reflecting prediction and prediction error), and subsequent higher-order error evaluation processes. The oN1 and oN2 of MA and VA were alike in amplitude, topography, and neural sources despite that the origin of the prediction stems from different brain areas (motor versus visual cortex). This suggests that MA and VA predictions activate a sensory template of the sound in auditory cortex. This article is part of a Special Issue entitled SI: Prediction and Attention.

[1]  C. C. Wood,et al.  Scalp distributions of event-related potentials: an ambiguity associated with analysis of variance models. , 1985, Electroencephalography and clinical neurophysiology.

[2]  E. Schafer,et al.  Self-Stimulation Alters Human Sensory Brain Responses , 1973, Science.

[3]  Luc H. Arnal,et al.  Transitions in neural oscillations reflect prediction errors generated in audiovisual speech , 2011, Nature Neuroscience.

[4]  R. Hari,et al.  Suppressed responses to self-triggered sounds in the human auditory cortex. , 2004, Cerebral cortex.

[5]  K. R. Ridderinkhof,et al.  Electrophysiological correlates of anterior cingulate function in a go/no-go task: Effects of response conflict and trial type frequency , 2003, Cognitive, affective & behavioral neuroscience.

[6]  J. Mäkelä,et al.  Human auditory cortex is activated by omissions of auditory stimuli , 1997, Brain Research.

[7]  N. Yeung,et al.  Dissociable correlates of response conflict and error awareness in error-related brain activity , 2011, Neuropsychologia.

[8]  Luc H. Arnal,et al.  Dual Neural Routing of Visual Facilitation in Speech Processing , 2009, The Journal of Neuroscience.

[9]  E. Bullmore,et al.  Response amplification in sensory-specific cortices during crossmodal binding. , 1999, Neuroreport.

[10]  Erich Schröger,et al.  I know what is missing here: electrophysiological prediction error signals elicited by omissions of predicted ”what” but not ”when” , 2013, Front. Hum. Neurosci..

[11]  E. Schröger,et al.  Hearing Silences: Human Auditory Processing Relies on Preactivation of Sound-Specific Brain Activity Patterns , 2013, The Journal of Neuroscience.

[12]  Shiwei Jia,et al.  Detecting perceptual conflict by the feedback-related negativity in brain potentials , 2007, Neuroreport.

[13]  J. Polich.  Updating P 300 : An Integrative Theory of P 3 a and P 3 b , 2009 .

[14]  D. Wolpert,et al.  Why can't you tickle yourself? , 2000, Neuroreport.

[15]  Luc H. Arnal,et al.  Cortical oscillations and sensory predictions , 2012, Trends in Cognitive Sciences.

[16]  J. Decety,et al.  From the perception of action to the understanding of intention , 2001, Nature reviews. Neuroscience.

[17]  Klucharev Vasily,et al.  Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception. , 2003 .

[18]  A. Fort,et al.  Bimodal speech: early suppressive visual effects in human auditory cortex , 2004, The European journal of neuroscience.

[19]  M. Sams,et al.  Primary auditory cortex activation by visual speech: an fMRI study at 3 T , 2005, Neuroreport.

[20]  J. Ford,et al.  Fine-tuning of auditory cortex during speech production. , 2005, Psychophysiology.

[21]  Jonathan D. Cohen,et al.  The neural basis of error detection: conflict monitoring and the error-related negativity. , 2004, Psychological review.

[22]  J. Vroomen,et al.  Electrophysiological correlates of predictive coding of auditory location in the perception of natural audiovisual events , 2012, Front. Integr. Neurosci..

[23]  Jeffery A. Jones,et al.  Neural processes underlying perceptual enhancement by visual speech gestures , 2003, Neuroreport.

[24]  J. Polich.  Updating P300: An integrative theory of P3a and P3b , 2007, Clinical Neurophysiology.

[25]  A. Giraud,et al.  Implicit Multisensory Associations Influence Voice Recognition , 2006, PLoS biology.

[26]  T. M. Darcey,et al.  Responses of Human Auditory Association Cortex to the Omission of an Expected Acoustic Event , 2001, NeuroImage.

[27]  E. Donchin.  Presidential address, 1980. Surprise!...Surprise? , 1981, Psychophysiology.

[28]  H. Yabe,et al.  Temporal window of integration revealed by MMN to sound omission , 1997, Neuroreport.

[29]  John J. Foxe,et al.  Multisensory processing of naturalistic objects in motion: A high-density electrical mapping and source estimation study , 2007, NeuroImage.

[30]  E. Schröger,et al.  Suppression of the auditory N1 event-related potential component with unpredictable self-initiated tones: evidence for internal forward models with dynamic stimulation. , 2008, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[31]  Jean Vroomen,et al.  Visual Anticipatory Information Modulates Multisensory Interactions of Artificial Audiovisual Stimuli , 2010, Journal of Cognitive Neuroscience.

[32]  Karl J. Friston,et al.  A theory of cortical responses , 2005, Philosophical Transactions of the Royal Society B: Biological Sciences.

[33]  R. Näätänen,et al.  The mismatch negativity (MMN) in basic research of central auditory processing: A review , 2007, Clinical Neurophysiology.

[34]  E Donchin,et al.  A new method for off-line removal of ocular artifact. , 1983, Electroencephalography and clinical neurophysiology.

[35]  M. Giard,et al.  Auditory-Visual Integration during Multimodal Object Recognition in Humans: A Behavioral and Electrophysiological Study , 1999, Journal of Cognitive Neuroscience.

[36]  J. Hohnsbein,et al.  ERP components on reaction errors and their functional significance: a tutorial , 2000, Biological Psychology.

[37]  Mike Wendt,et al.  Disentangling Sequential Effects of Stimulus- and Response-related Conflict and Stimulus-Response Repetition using Brain Potentials , 2007, Journal of Cognitive Neuroscience.

[38]  E. Bullmore,et al.  Activation of auditory cortex during silent lipreading. , 1997, Science.

[39]  D. Guthrie,et al.  Significance testing of difference potentials. , 1991, Psychophysiology.

[40]  Jeroen J. Stekelenburg.  Deficient multisensory integration in schizophrenia: An event-related potential study , 2013 .

[41]  C. Michel,et al.  Noninvasive Localization of Electromagnetic Epileptic Activity. I. Method Descriptions and Simulations , 2004, Brain Topography.

[42]  Denis Brunet,et al.  Topographic ERP Analyses: A Step-by-Step Tutorial Review , 2008, Brain Topography.

[43]  R. Heuser.  Surprise, surprise , 2014, Catheterization and cardiovascular interventions : official journal of the Society for Cardiac Angiography & Interventions.

[44]  Jean Vroomen,et al.  Neural Correlates of Multisensory Integration of Ecologically Valid Audiovisual Events , 2007, Journal of Cognitive Neuroscience.

[45]  David Poeppel,et al.  Visual speech speeds up the neural processing of auditory speech. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[46]  E Donchin,et al.  The effects of temporal and event uncertainty in determining the waveforms of the auditory event related potential (ERP). , 1976, Psychophysiology.