Normative evidence accumulation in unpredictable environments

In our dynamic world, decisions about noisy stimuli can require temporal accumulation of evidence to identify steady signals, differentiation to detect unpredictable changes in those signals, or both. Normative models can account for learning in these environments but have not yet been applied to faster decision processes. We present a novel, normative formulation of adaptive learning models that forms decisions by acting as a leaky accumulator with non-absorbing bounds. These dynamics, derived for both discrete and continuous cases, depend on the expected rate of change of the statistics of the evidence and balance signal identification and change detection. We found that, for two different tasks, human subjects learned these expectations, albeit imperfectly, then used them to make decisions in accordance with the normative model. The results represent a unified, empirically supported account of decision-making in unpredictable environments that provides new insights into the expectation-driven dynamics of the underlying neural signals. DOI: http://dx.doi.org/10.7554/eLife.08825.001

[1]  Philip L. Smith Bloch's law predictions from diffusion process models of detection , 1998 .

[2]  R. Duncan Luce,et al.  Response Times: Their Role in Inferring Elementary Mental Organization , 1986 .

[3]  Paul Cisek,et al.  Decision making by urgency gating: theory and experimental support. , 2012, Journal of neurophysiology.

[4]  Angela J. Yu,et al.  Uncertainty, Neuromodulation, and Attention , 2005, Neuron.

[5]  Scott D. Brown,et al.  Detecting and predicting changes , 2009, Cognitive Psychology.

[6]  P. Fearnhead,et al.  On‐line inference for multiple changepoint problems , 2007 .

[7]  Hatim A. Zariwala,et al.  Neural correlates, computation and behavioural impact of decision confidence , 2008, Nature.

[8]  Bingni W. Brunton,et al.  Distinct relationships of parietal and prefrontal cortices to evidence accumulation , 2014, Nature.

[9]  G. A. Barnard,et al.  Sequential Tests in Industrial Statistics , 1946 .

[10]  Maurice A. Smith,et al.  Environmental Consistency Determines the Rate of Motor Adaptation , 2014, Current Biology.

[11]  J. Gold,et al.  Representation of a perceptual decision in developing oculomotor commands , 2000, Nature.

[12]  Jonathan D. Cohen,et al.  The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced-choice tasks. , 2006, Psychological review.

[13]  Marius Usher,et al.  Using Time-Varying Evidence to Test Models of Decision Dynamics: Bounded Diffusion vs. the Leaky Competing Accumulator Model , 2012, Front. Neurosci..

[14]  Philip L. Smith Psychophysically principled models of visual simple reaction time. , 1995 .

[15]  Timothy E. J. Behrens,et al.  Learning the value of information in an uncertain world , 2007, Nature Neuroscience.

[16]  J. Townsend,et al.  Decision field theory: a dynamic-cognitive approach to decision making in an uncertain environment. , 1993, Psychological review.

[17]  Timothy D. Hanks,et al.  Elapsed Decision Time Affects the Weighting of Prior Probability in a Perceptual Decision Task , 2011, The Journal of Neuroscience.

[18]  M. Shadlen,et al.  The effect of stimulus strength on the speed and accuracy of a perceptual decision. , 2005, Journal of vision.

[19]  Joshua I. Gold,et al.  Bayesian Online Learning of the Hazard Rate in Change-Point Problems , 2010, Neural Computation.

[20]  Bingni W. Brunton,et al.  Rats and Humans Can Optimally Accumulate Evidence for Decision-Making , 2013, Science.

[21]  P. King-Smith,et al.  Efficient and unbiased modifications of the QUEST threshold method: Theory, simulations, experimental evaluation and practical implementation , 1994, Vision Research.

[22]  J. Movshon,et al.  The analysis of visual motion: a comparison of neuronal and psychophysical performance , 1992, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[23]  Ryan P. Adams,et al.  Bayesian Online Changepoint Detection , 2007, 0710.3742.

[24]  Alexandre Pouget,et al.  Optimal multisensory decision-making in a reaction-time task , 2014, eLife.

[25]  T Cohn,et al.  Detection of a luminance increment: effect of temporal uncertainty. , 1981, Journal of the Optical Society of America.

[26]  M. Shadlen,et al.  Representation of Confidence Associated with a Decision by Neurons in the Parietal Cortex , 2009, Science.

[27]  D. Broadbent,et al.  Some experiments bearing on the hypothesis that the visual system analyses spatial patterns in independent bands of spatial frequency , 1975, Vision Research.

[28]  Jonathan D. Cohen,et al.  Reward rate optimization in two-alternative decision making: empirical tests of theoretical predictions. , 2009, Journal of experimental psychology. Human perception and performance.

[29]  J I Gold,et al.  On diffusion processes with variable drift rates as models for decision making during learning , 2008, New journal of physics.

[30]  Leif H. Finkel,et al.  A Neural Implementation of the Kalman Filter , 2009, NIPS.

[31]  Robert C. Wilson,et al.  Rational regulation of learning dynamics by pupil–linked arousal systems , 2012, Nature Neuroscience.

[32]  David S. Touretzky,et al.  Advances in neural information processing systems 2 , 1989 .

[33]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[34]  Joseph W. Kable,et al.  Normative evidence accumulation in unpredictable environments , 2015, eLife.

[35]  Peter N. C. Mohr,et al.  Genetic variation in dopaminergic neuromodulation influences the ability to rapidly and flexibly adapt decisions , 2009, Proceedings of the National Academy of Sciences.

[36]  Paul R. Schrater,et al.  Mechanisms of visual motion detection , 2000, Nature Neuroscience.

[37]  J. Gold,et al.  Banburismus and the Brain Decoding the Relationship between Sensory Stimuli, Decisions, and Reward , 2002, Neuron.

[38]  A. Pouget,et al.  The Cost of Accumulating Evidence in Perceptual Decision Making , 2012, The Journal of Neuroscience.

[39]  S. Sternberg,et al.  Separate modifiability, mental modules, and the use of pure and composite measures to reveal them. , 2001, Acta psychologica.

[40]  R. Bogacz,et al.  The neural basis of the speed–accuracy tradeoff , 2010, Trends in Neurosciences.

[41]  D. M. Green,et al.  Signal detection theory and psychophysics , 1966 .

[42]  Joseph T. McGuire,et al.  Functionally Dissociable Influences on Learning Rate in a Dynamic Environment , 2014, Neuron.

[43]  Adam Kepecs,et al.  Seeing at a glance, smelling in a whiff: rapid forms of perceptual decision making , 2006, Nature Reviews Neuroscience.

[44]  Yoshiyuki Sato,et al.  How much to trust the senses: likelihood learning. , 2014, Journal of vision.

[45]  C. W. G Clifford,et al.  Fundamental mechanisms of visual motion detection: models, cells and functions , 2002, Progress in Neurobiology.

[46]  Timothy E. J. Behrens,et al.  Dissociable effects of surprise and model update in parietal and anterior cingulate cortex , 2013, Proceedings of the National Academy of Sciences.

[47]  J. Ditterich Evidence for time‐variant decision making , 2006, The European journal of neuroscience.

[48]  Philip L. Smith,et al.  Psychology and neurobiology of simple decisions , 2004, Trends in Neurosciences.

[49]  M. Shadlen,et al.  Response of Neurons in the Lateral Intraparietal Area during a Combined Visual Discrimination Reaction Time Task , 2002, The Journal of Neuroscience.

[50]  J. Gold,et al.  Neural computations that underlie decisions about sensory stimuli , 2001, Trends in Cognitive Sciences.

[51]  Kurt A. Thoroughman,et al.  Trial-by-trial transformation of error into sensorimotor adaptation changes with environmental dynamics. , 2007, Journal of neurophysiology.

[52]  Sophie Deneve,et al.  Making Decisions with Unknown Sensory Reliability , 2012, Front. Neurosci..

[53]  J. Nachmias,et al.  Masking by spatially-modulated gratings , 1983, Vision Research.

[54]  Wei Ji Ma,et al.  Neural coding of uncertainty and probability. , 2014, Annual review of neuroscience.

[55]  James L. McClelland,et al.  The time course of perceptual choice: the leaky, competing accumulator model. , 2001, Psychological review.

[56]  J. Stoyanov The Oxford Handbook of Nonlinear Filtering , 2012 .

[57]  Alʹbert Nikolaevich Shiri︠a︡ev,et al.  Statistics of random processes , 1977 .

[58]  J. Andel Sequential Analysis , 2022, The SAGE Encyclopedia of Research Design.

[59]  H. Heyer Statistics of random processes I: General theory , 1983 .

[60]  A. M. Turing,et al.  Studies in the History of Probability and Statistics. XXXVII , 1979 .

[61]  D. Wolpert,et al.  Changing your mind: a computational mechanism of vacillation , 2009, Nature.

[62]  Casimir J. H. Ludwig,et al.  The Temporal Impulse Response Underlying Saccadic Decisions , 2005, The Journal of Neuroscience.

[63]  A. Watson,et al.  Quest: A Bayesian adaptive psychometric method , 1983, Perception & psychophysics.

[64]  Nasser M. Nasrabadi,et al.  Pattern Recognition and Machine Learning , 2006, Technometrics.

[65]  Preeti Verghese,et al.  PII: S0042-6989(98)00033-9 , 1998 .

[66]  Christian K. Machens,et al.  Predictive Coding of Dynamical Variables in Balanced Spiking Networks , 2013, PLoS Comput. Biol..

[67]  H. Seal Studies in the history of probability and statistics , 1977 .

[68]  M. Zakai The Optimal Filtering of Markov Jump Processes in Additive White Noise , 2009 .

[69]  Robert C. Wilson,et al.  An Approximately Bayesian Delta-Rule Model Explains the Dynamics of Belief Updating in a Changing Environment , 2010, The Journal of Neuroscience.

[70]  J. Gold,et al.  The Basal Ganglia’s Contributions to Perceptual Decision Making , 2013, Neuron.

[71]  Joshua I. Gold,et al.  A Mixture of Delta-Rules Approximation to Bayesian Inference in Change-Point Problems , 2013, PLoS Comput. Biol..

[72]  Radford M. Neal Pattern Recognition and Machine Learning , 2007, Technometrics.

[73]  J. Gold,et al.  The neural basis of decision making. , 2007, Annual review of neuroscience.

[74]  R. Carpenter,et al.  The influence of urgency on decision time , 2000, Nature Neuroscience.

[75]  P. Cisek,et al.  Decisions in Changing Conditions: The Urgency-Gating Model , 2009, The Journal of Neuroscience.

[76]  Marius Usher,et al.  The Timescale of Perceptual Evidence Integration Can Be Adapted to the Environment , 2013, Current Biology.

[77]  Neil A. Macmillan,et al.  Detection Theory: A User's Guide , 1991 .

[78]  M. Shadlen,et al.  Neural Activity in Macaque Parietal Cortex Reflects Temporal Integration of Visual Motion Signals during Perceptual Decision Making , 2005, The Journal of Neuroscience.