Single Trial Plasticity in Evidence Accumulation Underlies Rapid Recalibration to Asynchronous Audiovisual Speech

Asynchronous arrival of audiovisual information at the peripheral sensory organs is a ubiquitous property of signals in the natural environment due to differences in the propagation time of light and sound. As these cues are constantly changing their distance from the observer, rapid adaptation to asynchronies is crucial for their appropriate integration. We investigated the neural basis of rapid recalibration to asynchronous audiovisual speech in humans using a combination of psychophysics, drift diffusion modeling, and electroencephalography (EEG). Consistent with previous reports, we found that perception of audiovisual temporal synchrony depends on the temporal ordering of the previous trial. Drift diffusion modelling indicated that this recalibration effect is well accounted for by changes in the rate of evidence accumulation (i.e. drift rate). Neural responses as indexed via evoked potentials were similarly found to vary based on the temporal ordering of the previous trial. Within and across subject correlations indicated that the observed changes in drift rate and the modulation of evoked potential magnitude were related. These results indicate that the rate and direction of evidence accumulation are affected by immediate sensory history and that these changes contribute to single trial recalibration to audiovisual temporal asynchrony.

[1]  T. Sejnowski,et al.  Removing electroencephalographic artifacts by blind source separation. , 2000, Psychophysiology.

[2]  Matthew A. De Niear,et al.  Atypical rapid audio‐visual temporal recalibration in autism spectrum disorders , 2017, Autism research : official journal of the International Society for Autism Research.

[3]  Erik Van der Burg,et al.  Rapid, generalized adaptation to asynchronous audiovisual speech , 2015, Proceedings of the Royal Society B: Biological Sciences.

[4]  W. H. Sumby,et al.  Visual contribution to speech intelligibility in noise , 1954 .

[5]  D. Poeppel,et al.  Temporal window of integration in auditory-visual speech perception , 2007, Neuropsychologia.

[6]  J. Fermaglich.  Electric Fields of the Brain: The Neurophysics of EEG , 1982 .

[7]  A. Voss,et al.  Interpreting the parameters of the diffusion model: An empirical validation , 2004, Memory & cognition.

[8]  A. Voss,et al.  Diffusion models in experimental psychology: a practical introduction. , 2013, Experimental psychology.

[9]  R. Oostenveld,et al.  Nonparametric statistical testing of EEG- and MEG-data , 2007, Journal of Neuroscience Methods.

[10]  Constant D. Beugré,et al.  The neural basis of decision making , 2018 .

[11]  Robert Oostenveld,et al.  FieldTrip: Open Source Software for Advanced Analysis of MEG, EEG, and Invasive Electrophysiological Data , 2010, Comput. Intell. Neurosci..

[12]  F. Perrin,et al.  Mapping of scalp potentials by surface spline interpolation. , 1987, Electroencephalography and clinical neurophysiology.

[13]  Francis Tuerlinckx,et al.  Diffusion model analysis with MATLAB: A DMAT primer , 2008, Behavior research methods.

[14]  D. Alais,et al.  Rapid Recalibration to Audiovisual Asynchrony , 2013, The Journal of Neuroscience.

[15]  S. Kelly,et al.  A supramodal accumulation-to-bound signal that determines perceptual decisions in humans , 2012, Nature Neuroscience.

[16]  T. Sejnowski,et al.  Removal of eye activity artifacts from visual event-related potentials in normal and clinical subjects , 2000, Clinical Neurophysiology.

[17]  M. Wallace,et al.  Event Related Potentials Index Rapid Recalibration to Audiovisual Temporal Asynchrony , 2017, Front. Integr. Neurosci..

[18]  M. Shadlen,et al.  Neural Activity in Macaque Parietal Cortex Reflects Temporal Integration of Visual Motion Signals during Perceptual Decision Making , 2005, The Journal of Neuroscience.

[19]  S. Nishida,et al.  Recalibration of audiovisual simultaneity , 2004, Nature Neuroscience.

[20]  Leslie G. Ungerleider,et al.  The neural systems that mediate human perceptual decision making , 2008, Nature Reviews Neuroscience.

[21]  S. Kelly,et al.  The classic P300 encodes a build‐to‐threshold decision variable , 2015, The European journal of neuroscience.

[22]  Matthew W. Mosconi,et al.  Reduced behavioral flexibility in autism spectrum disorders. , 2013, Neuropsychology.

[23]  David Alais,et al.  Audiovisual temporal recalibration occurs independently at two different time scales , 2015, Scientific Reports.

[24]  Albert R. Powers,et al.  Perceptual Training Narrows the Temporal Window of Multisensory Binding , 2009, The Journal of Neuroscience.

[25]  J. Bizley.  The Neural Bases of Multisensory Processes , 2011 .

[26]  Mark T. Wallace,et al.  Integration and Temporal Processing of Asynchronous Audiovisual Speech , 2018, Journal of Cognitive Neuroscience.

[27]  Arnaud Delorme,et al.  EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis , 2004, Journal of Neuroscience Methods.

[28]  P. Wang,et al.  P300 amplitude and latency in autism spectrum disorder: a meta-analysis , 2017, European Child & Adolescent Psychiatry.

[29]  Jennifer K Bizley,et al.  Where are multisensory signals combined for perceptual decision-making? , 2016, Current Opinion in Neurobiology.

[30]  John J. Foxe,et al.  Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments. , 2006, Cerebral cortex.

[31]  Alan A. Stocker,et al.  Sensory Adaptation within a Bayesian Framework for Perception , 2005, NIPS.

[32]  D. Angelaki,et al.  A computational perspective on autism , 2015, Proceedings of the National Academy of Sciences.

[33]  S. Kelly,et al.  The neural processes underlying perceptual decision making in humans: Recent progress and future directions , 2015, Journal of Physiology-Paris.

[34]  J. Gold,et al.  The neural basis of decision making. , 2007, Annual review of neuroscience.

[35]  T. Womelsdorf,et al.  Attentional Stimulus Selection through Selective Synchronization between Monkey Visual Areas , 2012, Neuron.

[36]  P. Bertelson,et al.  Recalibration of temporal order perception by exposure to audio-visual asynchrony. , 2004, Brain research. Cognitive brain research.

[37]  Alexandre Gramfort,et al.  Encoding of event timing in the phase of neural oscillations , 2014, NeuroImage.

[38]  S. Kelly,et al.  Internal and External Influences on the Rate of Sensory Evidence Accumulation in the Human Brain , 2013, The Journal of Neuroscience.

[39]  A. Borst.  Seeing smells: imaging olfactory learning in bees , 1999, Nature Neuroscience.

[40]  S. Baron-Cohen,et al.  Atypical EEG complexity in autism spectrum conditions: A multiscale entropy analysis , 2011, Clinical Neurophysiology.

[41]  Néstor Parga,et al.  Internal signal correlates neural populations and biases perceptual decision reports , 2012, Proceedings of the National Academy of Sciences.

[42]  J. Vroomen,et al.  Neural correlates of motor–sensory temporal recalibration , 2011, Brain Research.

[43]  Francis Tuerlinckx,et al.  Fitting the ratcliff diffusion model to experimental data , 2007, Psychonomic bulletin & review.

[44]  Markus Siegel,et al.  Cortical information flow during flexible sensorimotor decisions , 2015, Science.

[45]  Daniel P. Newman,et al.  Target Selection Signals Influence Perceptual Decisions by Modulating the Onset and Rate of Evidence Accumulation , 2016, Current Biology.

[46]  M. Wallace,et al.  Multisensory temporal integration: task and stimulus dependencies , 2013, Experimental Brain Research.

[47]  Jeffrey N. Rouder,et al.  Modeling Response Times for Two-Choice Decisions , 1998 .

[48]  Roger Ratcliff,et al.  The Diffusion Decision Model: Theory and Data for Two-Choice Decision Tasks , 2008, Neural Computation.

[49]  D. Burr,et al.  No rapid audiovisual recalibration in adults on the autism spectrum , 2016, Scientific Reports.

[50]  W. Stone,et al.  Neural Correlates of Sensory Hyporesponsiveness in Toddlers at High Risk for Autism Spectrum Disorder , 2017, Journal of Autism and Developmental Disorders.