Recalibration of the Multisensory Temporal Window of Integration Results from Changing Task Demands

The notion of the temporal window of integration, when applied in a multisensory context, refers to the breadth of the interval across which the brain perceives two stimuli from different sensory modalities as synchronous. It maintains a unitary perception of multisensory events despite physical and biophysical timing differences between the senses. The boundaries of the window can be influenced by attention and past sensory experience. Here we examined whether task demands could also influence the multisensory temporal window of integration. We varied the stimulus onset asynchrony between simple, short-lasting auditory and visual stimuli while participants performed two tasks in separate blocks: a temporal order judgment task that required the discrimination of subtle auditory-visual asynchronies, and a reaction time task to the first incoming stimulus irrespective of its sensory modality. We defined the temporal window of integration as the range of stimulus onset asynchronies where performance was below 75% in the temporal order judgment task, as well as the range of stimulus onset asynchronies where responses showed multisensory facilitation (race model violation) in the reaction time task. In 5 of 11 participants, we observed audio-visual stimulus onset asynchronies where reaction time was significantly accelerated (indicating successful integration in this task) while performance was accurate in the temporal order judgment task (indicating successful segregation in that task). This dissociation suggests that in some participants, the boundaries of the temporal window of integration can adaptively recalibrate in order to optimize performance according to specific task demands.

[1]  I. Hirsh,et al.  Perceived order in different sense modalities. , 1961, Journal of experimental psychology.

[2]  J. Rutschmann,et al.  Perception of Temporal Order of Stimuli Differing in Sense Mode and Simple Reaction Time , 1964, Perceptual and motor skills.

[3]  R. L. Knoll,et al.  The Perception of Temporal Order: Fundamental Issues and a General Model , 1973 .

[4]  N. F. Dixon,et al.  The Detection of Auditory Visual Desynchrony , 1980, Perception.

[5]  Jeff Miller,et al.  Divided attention: Evidence for coactivation with redundant signals , 1982, Cognitive Psychology.

[6]  Jeff Miller,et al.  Timecourse of coactivation in bimodal divided attention , 1986, Perception & psychophysics.

[7]  B. Stein,et al.  Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors , 1987, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[8]  C W Eriksen,et al.  A source of error in attempts to distinguish coactivation from separate activation in the perception of redundant targets , 1988, Perception & psychophysics.

[9]  Piotr Jaśkowski,et al.  Temporal-order judgments and reaction time for stimuli of different modalities , 1990, Psychological research.

[10]  Jeff Miller,et al.  Bias produced by fast guessing in distribution-based tests of race models , 1991, Perception & Psychophysics.

[11]  P. Jaśkowski,et al.  Simple Reaction Time and Perception of Temporal Order: Dissociations and Hypotheses , 1996, Perceptual and motor skills.

[12]  M. Wallace,et al.  Representation and integration of multiple sensory inputs in primate superior colliculus. , 1996, Journal of neurophysiology.

[13]  M. Giard,et al.  Auditory-Visual Integration during Multimodal Object Recognition in Humans: A Behavioral and Electrophysiological Study , 1999, Journal of Cognitive Neuroscience.

[14]  H. Strasburger,et al.  Fitting the psychometric function , 1999, Perception & psychophysics.

[15]  C. Spence,et al.  Multisensory prior entry. , 2001, Journal of experimental psychology. General.

[16]  S. Iversen,et al.  Detection of Audio-Visual Integration Sites in Humans by Application of Electrophysiological Criteria to the BOLD Effect , 2001, NeuroImage.

[17]  M. Hallett,et al.  Neural Correlates of Auditory–Visual Stimulus Onset Asynchrony Detection , 2001, The Journal of Neuroscience.

[18]  James V. Stone,et al.  When is now? Perception of simultaneity , 2001, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[19]  F A Wichmann,et al.  Ning for Helpful Comments and Suggestions. This Paper Benefited Con- Siderably from Conscientious Peer Review, and We Thank Our Reviewers the Psychometric Function: I. Fitting, Sampling, and Goodness of Fit , 2001 .

[20]  John J. Foxe,et al.  Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. , 2002, Brain research. Cognitive brain research.

[21]  Radford M. Neal.  Slice Sampling , 2003, The Annals of Statistics.

[22]  M. Hallett,et al.  Neural correlates of cross-modal binding , 2003, Nature Neuroscience.

[23]  C. Spence,et al.  Multisensory Integration: Maintaining the Perception of Synchrony , 2003, Current Biology.

[24]  C. S. Green,et al.  Action video game modifies visual selective attention , 2003, Nature.

[25]  S. Walker.  Invited comment on the paper "Slice Sampling" by Radford Neal , 2003 .

[26]  C. Spence,et al.  Audiovisual temporal order judgments , 2003, Experimental Brain Research.

[27]  Jeff Miller,et al.  Exaggerated redundancy gain in the split brain: A hemispheric coactivation account , 2004, Cognitive Psychology.

[28]  A. Diederich,et al.  Bimodal and trimodal multisensory enhancement: Effects of stimulus onset and intensity on reaction time , 2004, Perception & psychophysics.

[29]  E. Schröger,et al.  Psychophysics beyond sensation : laws and invariants of human cognition , 2004 .

[30]  M. Wallace.  The Development of Multisensory Integration. , 2004 .

[31]  S. Nishida,et al.  Recalibration of audiovisual simultaneity , 2004, Nature Neuroscience.

[32]  P. Bertelson,et al.  Recalibration of temporal order perception by exposure to audio-visual asynchrony. , 2004, Brain research. Cognitive brain research.

[33]  O. Neumann,et al.  Timing of perception and perception of time , 2004 .

[34]  W. David Hairston,et al.  Altered temporal profile of visual–auditory multisensory interactions in dyslexia , 2005, Experimental Brain Research.

[35]  W. Hairston,et al.  Aspects of Multisensory Perception: The Integration of Visual and Auditory Information in Musical Experiences , 2005, Annals of the New York Academy of Sciences.

[36]  John J. Foxe,et al.  Grabbing your ear: rapid auditory-somatosensory multisensory interactions in low-level sensory cortices are not constrained by stimulus alignment. , 2005, Cerebral cortex.

[37]  C. Spence,et al.  Audiovisual prior entry , 2003, Neuroscience Letters.

[38]  Frank Jäkel,et al.  Bayesian inference for psychometric functions. , 2005, Journal of vision.

[39]  C. Spence,et al.  Audio-visual simultaneity judgments , 2005, Perception & psychophysics.

[40]  Marty G. Woldorff,et al.  Selective Attention and Multisensory Integration: Multiple Phases of Effects on the Evoked Brain Activity , 2005, Journal of Cognitive Neuroscience.

[41]  W. Schwarz.  On the relationship between the redundant signals effect and temporal order judgments: parametric data and a new model. , 2006, Journal of experimental psychology. Human perception and performance.

[42]  John J. Foxe,et al.  Audio-visual multisensory integration in superior parietal lobule revealed by human intracranial recordings. , 2006, Journal of neurophysiology.

[43]  John J. Foxe,et al.  Oscillatory beta activity predicts response speed during a multisensory audiovisual reaction time task: a high-density electrical mapping study. , 2005, Cerebral cortex.

[44]  A. Diederich,et al.  The race model inequality: interpreting a geometric measure of the amount of violation. , 2006, Psychological review.

[45]  C. Spence,et al.  Audiovisual synchrony perception for music, speech, and object actions , 2006, Brain Research.

[46]  Jeff Miller,et al.  Dissociations between reaction times and temporal order judgments: a diffusion model approach. , 2006, Journal of experimental psychology. Human perception and performance.

[47]  D. Poeppel,et al.  Temporal window of integration in auditory-visual speech perception , 2007, Neuropsychologia.

[48]  J. Rieger,et al.  Audiovisual Temporal Correspondence Modulates Human Multisensory Superior Temporal Sulcus Plus Primary Sensory Cortices , 2007, The Journal of Neuroscience.

[49]  A. Gorea,et al.  Temporal order judgment and simple reaction times: evidence for a common processing system. , 2007, Journal of vision.

[50]  C. Spence,et al.  Audiovisual temporal adaptation of speech: temporal order versus simultaneity judgments , 2008, Experimental Brain Research.

[51]  Rolf Ulrich,et al.  Testing the race model inequality: An algorithm and computer programs , 2007, Behavior research methods.

[52]  Jeff Miller,et al.  Systematic biases and Type I error accumulation in tests of the race model inequality , 2007, Behavior research methods.

[53]  J. Juola,et al.  Audiovisual synchrony and temporal order judgments: Effects of experimental method and stimulus type , 2008, Perception & psychophysics.

[54]  Matthias Gondan,et al.  Testing the race inequality: A simple correction procedure for fast guesses , 2008 .

[55]  John J. Foxe,et al.  Changes in effective connectivity of human superior parietal lobule under multisensory and unisensory stimulation , 2008, The European journal of neuroscience.

[56]  S. Nishida,et al.  Audio–tactile superiority over visuo–tactile and audio–visual combinations in the temporal resolution of synchrony perception , 2009, Experimental Brain Research.

[57]  A. Diederich,et al.  Crossmodal interaction in speeded responses: time window of integration model. , 2009, Progress in brain research.

[58]  Davide Rocchesso,et al.  Multisensory integration of drumming actions: musical expertise affects perceived audiovisual asynchrony , 2009, Experimental Brain Research.

[59]  Lars T. Boenke,et al.  Stimulus duration influences perceived simultaneity in audiovisual temporal-order judgment , 2009, Experimental Brain Research.

[60]  Matthias Gondan,et al.  Testing the race model inequality in redundant stimuli with variable onset asynchrony. , 2009, Journal of experimental psychology. Human perception and performance.

[61]  Micah M. Murray,et al.  Early, Low-Level Auditory-Somatosensory Multisensory Interactions Impact Reaction Time Speed , 2009, Front. Integr. Neurosci..

[62]  Albert R. Powers,et al.  Perceptual Training Narrows the Temporal Window of Multisensory Binding , 2009, The Journal of Neuroscience.

[63]  Hans Colonius,et al.  The race model inequality for censored reaction time distributions , 2010, Attention, perception & psychophysics.

[64]  W. Stone,et al.  An extended multisensory temporal binding window in autism spectrum disorders , 2010, Experimental Brain Research.

[65]  S. Mitroff,et al.  Video game players show more precise multisensory temporal processing abilities , 2010, Attention, perception & psychophysics.

[66]  Neil W. Roach,et al.  Attention regulates the plasticity of multisensory timing , 2010, The European journal of neuroscience.

[67]  Matthias Gondan,et al.  A permutation test for the race model inequality , 2010, Behavior research methods.

[68]  Fabien Scalzo,et al.  Reducing backward masking through action game training. , 2010, Journal of vision.

[69]  C. Spence,et al.  Prior-entry: A review , 2010, Consciousness and Cognition.

[70]  J. Vroomen,et al.  Perception of intersensory synchrony: A tutorial review , 2010, Attention, perception & psychophysics.

[71]  John J. Foxe,et al.  The development of audiovisual multisensory integration across childhood and early adolescence: a high-density electrical mapping study. , 2011, Cerebral cortex.

[72]  John J. Foxe,et al.  Multisensory interactions in early evoked brain activity follow the principle of inverse effectiveness , 2011, NeuroImage.

[73]  Albert R. Powers,et al.  Binding of sights and sounds: Age-related changes in multisensory temporal processing , 2011, Neuropsychologia.

[74]  Mark T. Wallace,et al.  Altered Auditory and Multisensory Temporal Processing in Autism Spectrum Disorders , 2011, Front. Integr. Neurosci..

[75]  K. Krumbholz,et al.  Evidence for multisensory integration in the elicitation of prior entry by bimodal cues , 2012, Experimental Brain Research.

[76]  Albert R. Powers,et al.  Neural Correlates of Multisensory Perceptual Learning , 2012, The Journal of Neuroscience.

[77]  Hans-Jochen Heinze,et al.  Coding of multisensory temporal patterns in human superior temporal sulcus , 2012, Front. Integr. Neurosci..

[78]  M. Wallace,et al.  Individual differences in the multisensory temporal binding window predict susceptibility to audiovisual illusions. , 2012, Journal of experimental psychology. Human perception and performance.

[79]  John J. Foxe,et al.  The development of multisensory integration in high-functioning autism: high-density electrical mapping and psychophysical measures reveal impairments in the processing of audiovisual inputs. , 2013, Cerebral cortex.

[80]  Jens-Max Hopf,et al.  Task-demands and audio-visual stimulus configurations modulate neural activity in the human thalamus , 2013, NeuroImage.