Optimal multisensory decision-making in a reaction-time task

Humans and animals can integrate sensory evidence from various sources to make decisions in a statistically near-optimal manner, provided that the stimulus presentation time is fixed across trials. Little is known about whether optimality is preserved when subjects can choose when to make a decision (reaction-time task), nor when sensory inputs have time-varying reliability. Using a reaction-time version of a visual/vestibular heading discrimination task, we show that behavior is clearly sub-optimal when quantified with traditional optimality metrics that ignore reaction times. We created a computational model that accumulates evidence optimally across both cues and time, and trades off accuracy with decision speed. This model quantitatively explains subjects's choices and reaction times, supporting the hypothesis that subjects do, in fact, accumulate evidence optimally over time and across sensory modalities, even when the reaction time is under the subject's control. DOI: http://dx.doi.org/10.7554/eLife.03005.001

[1]  Jonathan D. Cohen,et al.  The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced-choice tasks. , 2006, Psychological review.

[2]  E. Salinas,et al.  Perceptual decision making in less than 30 milliseconds , 2010, Nature Neuroscience.

[3]  J. Goldberg,et al.  Physiology of peripheral neurons innervating otolith organs of the squirrel monkey. III. Response dynamics. , 1976, Journal of neurophysiology.

[4]  Felix Wichmann,et al.  The psychometric function: II. Bootstrap-based confidence intervals and sampling , 2001, Perception & psychophysics.

[5]  M. Shadlen,et al.  The effect of stimulus strength on the speed and accuracy of a perceptual decision. , 2005, Journal of vision.

[6]  E. A. Whitchurch,et al.  Combined auditory and visual stimuli facilitate head saccades in the barn owl (Tyto alba). , 2006, Journal of neurophysiology.

[7]  Kenneth H Britten,et al.  Area MST and heading perception in macaque monkeys. , 2002, Cerebral cortex.

[8]  Karl J. Friston,et al.  Bayesian model selection for group studies , 2009, NeuroImage.

[9]  Dora E Angelaki,et al.  Visual and Nonvisual Contributions to Three-Dimensional Heading Selectivity in the Medial Superior Temporal Area , 2006, The Journal of Neuroscience.

[10]  James J. Clark,et al.  Data Fusion for Sensory Information Processing Systems , 1990 .

[11]  Philip L. Smith,et al.  A comparison of sequential sampling models for two-choice reaction time. , 2004, Psychological review.

[12]  G. DeAngelis,et al.  Neural correlates of multisensory cue integration in macaque MSTd , 2008, Nature Neuroscience.

[13]  J. Movshon,et al.  The statistical reliability of signals in single neurons in cat and monkey visual cortex , 1983, Vision Research.

[14]  Emilio Salinas,et al.  Perceptual Decision Making , 2014, Encyclopedia of Computational Neuroscience.

[15]  Christopher R Fetsch,et al.  Dynamic Reweighting of Visual and Vestibular Cues during Self-Motion Perception , 2009, The Journal of Neuroscience.

[16]  Maninder K. Kahlon,et al.  Visual Motion Analysis for Pursuit Eye Movements in Area MT of Macaque Monkeys , 1999, The Journal of Neuroscience.

[17]  G. DeAngelis,et al.  Representation of Vestibular and Visual Cues to Self-Motion in Ventral Intraparietal Cortex , 2011, The Journal of Neuroscience.

[18]  Yong Gu,et al.  Spatiotemporal properties of vestibular responses in area MSTd. , 2010, Journal of neurophysiology.

[19]  Robert J. van Beers,et al.  How humans combine simultaneous proprioceptive and visual position information , 1996, Experimental Brain Research.

[20]  Jeff Miller,et al.  Divided attention: Evidence for coactivation with redundant signals , 1982, Cognitive Psychology.

[21]  Wei Ji Ma,et al.  Bayesian inference with probabilistic population codes , 2006, Nature Neuroscience.

[22]  G. DeAngelis,et al.  Multisensory Integration in Macaque Visual Cortex Depends on Cue Reliability , 2008, Neuron.

[23]  H Colonius,et al.  A two-stage model for visual-auditory interaction in saccadic latencies , 2001, Perception & psychophysics.

[24]  Yong Gu,et al.  Decoding of MSTd Population Activity Accounts for Variations in the Precision of Heading Perception , 2010, Neuron.

[25]  Thomas U. Otto,et al.  Noise and Correlations in Parallel Perceptual Decision Making , 2011, Current Biology.

[26]  Philip L. Smith,et al.  Stochastic Dynamic Models of Response Time and Accuracy: A Foundational Primer. , 2000, Journal of mathematical psychology.

[27]  Yong Gu,et al.  Causal Links between Dorsal Medial Superior Temporal Area Neurons and Multisensory Heading Perception , 2012, The Journal of Neuroscience.

[28]  D. Raab.  DIVISION OF PSYCHOLOGY: STATISTICAL FACILITATION OF SIMPLE REACTION TIMES* , 1962 .

[29]  J. Townsend,et al.  A theory of interactive parallel processing: new capacity measures and predictions for a response time inequality series. , 2004, Psychological review.

[30]  D. Raab.  Statistical facilitation of simple reaction times. , 1962, Transactions of the New York Academy of Sciences.

[31]  A J Van Opstal,et al.  Auditory-visual interactions subserving goal-directed saccades in a complex scene. , 2002, Journal of neurophysiology.

[32]  M. Ernst,et al.  Humans integrate visual and haptic information in a statistically optimal fashion , 2002, Nature.

[33]  S. Sterbing-D’Angelo,et al.  Behavioral/systems/cognitive Multisensory Space Representations in the Macaque Ventral Intraparietal Area , 2022 .

[34]  Timothy D. Hanks,et al.  Bounded Integration in Parietal Cortex Underlies Decisions Even When Viewing Duration Is Dictated by the Environment , 2008, The Journal of Neuroscience.

[35]  L. M. M.-T..  Theory of Probability , 1929, Nature.

[36]  Arnulf B. A. Graf,et al.  Decoding the activity of neuronal populations in macaque primary visual cortex , 2011, Nature Neuroscience.

[37]  M. Shadlen,et al.  A role for neural integrators in perceptual decision making. , 2003, Cerebral cortex.

[38]  Kenneth H Britten,et al.  Linear responses to stochastic motion signals in area MST. , 2007, Journal of neurophysiology.

[39]  M. Ibbotson,et al.  Comparing acceleration and speed tuning in macaque MT: physiology and modeling. , 2005, Journal of neurophysiology.

[40]  J. Goldberg,et al.  Physiology of peripheral neurons innervating otolith organs of the squirrel monkey. II. Directional selectivity and force-response relations. , 1976, Journal of neurophysiology.

[41]  K. Nagai,et al.  SCN output drives the autonomic nervous system: with special reference to the autonomic function related to the regulation of glucose metabolism. , 1996, Progress in brain research.

[42]  G. DeAngelis,et al.  A functional link between area MSTd and heading perception based on vestibular signals , 2007, Nature Neuroscience.

[43]  Roger Ratcliff,et al.  A Theory of Memory Retrieval. , 1978 .

[44]  Gregory C. DeAngelis,et al.  A Comparison of Vestibular Spatiotemporal Tuning in Macaque Parietoinsular Vestibular Cortex, Ventral Intraparietal Area, and Medial Superior Temporal Area , 2011, The Journal of Neuroscience.

[45]  Timothy D. Hanks,et al.  Probabilistic Population Codes for Bayesian Decision Making , 2008, Neuron.

[46]  Athanasios Papoulis,et al.  Probability, Random Variables and Stochastic Processes , 1965 .

[47]  G R Grice,et al.  Combination rule for redundant information in reaction time tasks with divided attention , 1984, Perception & psychophysics.

[48]  Donald Laming,et al.  Information theory of choice-reaction times , 1968 .

[49]  J. Saunders,et al.  Do humans optimally integrate stereo and texture information for judgments of surface slant? , 2003, Vision Research.

[50]  Robert A Jacobs,et al.  Bayesian integration of visual and auditory signals for spatial localization. , 2003, Journal of the Optical Society of America. A, Optics, image science, and vision.

[51]  Christopher R Fetsch,et al.  Neural correlates of reliability-based cue weighting during multisensory integration , 2011, Nature Neuroscience.

[52]  S. Goodman,et al.  Toward Evidence-Based Medical Statistics. 2: The Bayes Factor , 1999, Annals of Internal Medicine.