Optimal multisensory decision-making 1 in a reaction-time task 2 3 Running title : Multisensory decision-making in reaction-time task 4 5

Humans and animals can integrate sensory evidence from various sources to make decisions in a statistically near-optimal manner, provided that the stimulus presentation time is fixed across trials. Little is known about whether optimality is preserved when subjects can choose when to make a decision (reaction-time task), nor when sensory inputs have time-varying reliability. Using a reaction-time version of a visual/vestibular heading discrimination task, we show that behavior is clearly sub-optimal when quantified with traditional optimality metrics that ignore reaction times. We created a computational model that accumulates evidence optimally across both cues and time, and trades off accuracy with decision speed. This model quantitatively explains subjects' choices and reaction times, supporting the hypothesis that subjects do, in fact, accumulate evidence optimally over time and across sensory modalities, even when the reaction time is under the subject's control. DRUGOWITSCH, Jan, et al. Optimal multisensory decision-making in a reaction-time task. eLife, 2014, vol. n/a, p. e03005 DOI : 10.7554/eLife.03005

[1]  Yong Gu,et al.  Causal Links between Dorsal Medial Superior Temporal Area Neurons and Multisensory Heading Perception , 2012, The Journal of Neuroscience.

[2]  Christopher R Fetsch,et al.  Neural correlates of reliability-based cue weighting during multisensory integration , 2011, Nature Neuroscience.

[3]  Thomas U. Otto,et al.  Noise and Correlations in Parallel Perceptual Decision Making , 2011, Current Biology.

[4]  G. DeAngelis,et al.  Representation of Vestibular and Visual Cues to Self-Motion in Ventral Intraparietal Cortex , 2011, The Journal of Neuroscience.

[5]  Gregory C. DeAngelis,et al.  A Comparison of Vestibular Spatiotemporal Tuning in Macaque Parietoinsular Vestibular Cortex, Ventral Intraparietal Area, and Medial Superior Temporal Area , 2011, The Journal of Neuroscience.

[6]  Arnulf B. A. Graf,et al.  Decoding the activity of neuronal populations in macaque primary visual cortex , 2011, Nature Neuroscience.

[7]  Yong Gu,et al.  Spatiotemporal properties of vestibular responses in area MSTd. , 2010, Journal of neurophysiology.

[8]  Yong Gu,et al.  Decoding of MSTd Population Activity Accounts for Variations in the Precision of Heading Perception , 2010, Neuron.

[9]  Christopher R Fetsch,et al.  Dynamic Reweighting of Visual and Vestibular Cues during Self-Motion Perception , 2009, The Journal of Neuroscience.

[10]  Karl J. Friston,et al.  Bayesian model selection for group studies , 2009, NeuroImage.

[11]  Timothy D. Hanks,et al.  Probabilistic Population Codes for Bayesian Decision Making , 2008, Neuron.

[12]  G. DeAngelis,et al.  Neural correlates of multisensory cue integration in macaque MSTd , 2008, Nature Neuroscience.

[13]  G. DeAngelis,et al.  Multisensory Integration in Macaque Visual Cortex Depends on Cue Reliability , 2008, Neuron.

[14]  Timothy D. Hanks,et al.  Bounded Integration in Parietal Cortex Underlies Decisions Even When Viewing Duration Is Dictated by the Environment , 2008, The Journal of Neuroscience.

[15]  Kenneth H Britten,et al.  Linear responses to stochastic motion signals in area MST. , 2007, Journal of neurophysiology.

[16]  G. DeAngelis,et al.  A functional link between area MSTd and heading perception based on vestibular signals , 2007, Nature Neuroscience.

[17]  Jonathan D. Cohen,et al.  The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced-choice tasks. , 2006, Psychological review.

[18]  E. A. Whitchurch,et al.  Combined auditory and visual stimuli facilitate head saccades in the barn owl (Tyto alba). , 2006, Journal of neurophysiology.

[19]  Dora E Angelaki,et al.  Visual and Nonvisual Contributions to Three-Dimensional Heading Selectivity in the Medial Superior Temporal Area , 2006, The Journal of Neuroscience.

[20]  M. Ibbotson,et al.  Comparing acceleration and speed tuning in macaque MT: physiology and modeling. , 2005, Journal of neurophysiology.

[21]  M. Shadlen,et al.  The effect of stimulus strength on the speed and accuracy of a perceptual decision. , 2005, Journal of vision.

[22]  J. Townsend,et al.  A theory of interactive parallel processing: new capacity measures and predictions for a response time inequality series. , 2004, Psychological review.

[23]  Philip L. Smith,et al.  A comparison of sequential sampling models for two-choice reaction time. , 2004, Psychological review.

[24]  J. Saunders,et al.  Do humans optimally integrate stereo and texture information for judgments of surface slant? , 2003, Vision Research.

[25]  M. Shadlen,et al.  A role for neural integrators in perceptual decision making. , 2003, Cerebral cortex.

[26]  Robert A Jacobs,et al.  Bayesian integration of visual and auditory signals for spatial localization. , 2003, Journal of the Optical Society of America. A, Optics, image science, and vision.

[27]  A J Van Opstal,et al.  Auditory-visual interactions subserving goal-directed saccades in a complex scene. , 2002, Journal of neurophysiology.

[28]  Kenneth H Britten,et al.  Area MST and heading perception in macaque monkeys. , 2002, Cerebral cortex.

[29]  M. Ernst,et al.  Humans integrate visual and haptic information in a statistically optimal fashion , 2002, Nature.

[30]  Felix Wichmann,et al.  The psychometric function: II. Bootstrap-based confidence intervals and sampling , 2001, Perception & psychophysics.

[31]  Philip L. Smith,et al.  Stochastic Dynamic Models of Response Time and Accuracy: A Foundational Primer. , 2000, Journal of mathematical psychology.

[32]  M. Moia,et al.  Subcutaneous Heparin for Deep Venous Thrombosis , 1999, Annals of Internal Medicine.

[33]  Maninder K. Kahlon,et al.  Visual Motion Analysis for Pursuit Eye Movements in Area MT of Macaque Monkeys , 1999, The Journal of Neuroscience.

[34]  K. H. Britten,et al.  Electrical microstimulation of cortical area MST biases heading perception in monkeys , 1998, Nature Neuroscience.

[35]  Robert J. van Beers,et al.  How humans combine simultaneous proprioceptive and visual position information , 1996, Experimental Brain Research.

[36]  James J. Clark,et al.  Data Fusion for Sensory Information Processing Systems , 1990 .

[37]  John G. Proakis,et al.  Probability, random variables and stochastic processes , 1985, IEEE Trans. Acoust. Speech Signal Process..

[38]  G R Grice,et al.  Combination rule for redundant information in reaction time tasks with divided attention , 1984, Perception & psychophysics.

[39]  J. Movshon,et al.  The statistical reliability of signals in single neurons in cat and monkey visual cortex , 1983, Vision Research.

[40]  Jeff Miller,et al.  Divided attention: Evidence for coactivation with redundant signals , 1982, Cognitive Psychology.

[41]  Roger Ratcliff,et al.  A Theory of Memory Retrieval. , 1978 .

[42]  J. Goldberg,et al.  Physiology of peripheral neurons innervating otolith organs of the squirrel monkey. III. Response dynamics. , 1976, Journal of neurophysiology.

[43]  Donald Laming,et al.  Information theory of choice-reaction times , 1968 .

[44]  D. Raab.  Statistical facilitation of simple reaction times. , 1962, Transactions of the New York Academy of Sciences.

[45]  A. N. Kolmogorov,et al.  Theory of Probability , 1929, Nature.

[46]  S. Sterbing-D’Angelo,et al.  Behavioral/systems/cognitive Multisensory Space Representations in the Macaque Ventral Intraparietal Area , 2022 .