Confluence of Timing and Reward Biases in Perceptual Decision-Making Dynamics

Although the decisions of our daily lives often occur in the context of temporal and reward structures, the impact of such regularities on decision-making strategy is poorly understood. Here, to explore how temporal and reward context modulate strategy, we trained 2 male rhesus monkeys to perform a novel perceptual decision-making task with asymmetric rewards and time-varying evidence reliability. To model the choice and response time patterns, we developed a computational framework for fitting generalized drift-diffusion models, which flexibly accommodate diverse evidence accumulation strategies. We found that a dynamic urgency signal and leaky integration, in combination with two independent forms of reward biases, best capture behavior. We also tested how temporal structure influences urgency by systematically manipulating the temporal structure of sensory evidence, and found that the time course of urgency was affected by temporal context. Overall, our approach identified key components of cognitive mechanisms for incorporating temporal and reward structure into decisions. SIGNIFICANCE STATEMENT In everyday life, decisions are influenced by many factors, including reward structures and stimulus timing. While reward and timing have been characterized in isolation, ecologically valid decision-making involves a multiplicity of factors acting simultaneously. This raises questions about whether the same decision-making strategy is used when these two factors are concurrently manipulated. To address these questions, we trained rhesus monkeys to perform a novel decision-making task with both reward asymmetry and temporal uncertainty. In order to understand their strategy and hint at its neural mechanisms, we used the new generalized drift diffusion modeling framework to model both reward and timing mechanisms. We found two of each reward and timing mechanisms are necessary to explain our data.

[1]  P. Cisek,et al.  The Basal Ganglia Do Not Select Reach Targets but Control the Urgency of Commitment , 2017, Neuron.

[2]  Donald Laming,et al.  Information theory of choice-reaction times , 1968 .

[3]  K. R. Ridderinkhof,et al.  Striatum and pre-SMA facilitate decision-making under time pressure , 2008, Proceedings of the National Academy of Sciences.

[4]  Tobias Teichert,et al.  Suboptimal Integration of Reward Magnitude and Prior Reward Likelihood in Categorical Decisions by Monkeys , 2010, Front. Neurosci..

[5]  Scott D. Brown,et al.  Discriminating evidence accumulation from urgency signals in speeded decision making. , 2015, Journal of neurophysiology.

[6]  Roozbeh Kiani,et al.  A neural mechanism of speed-accuracy tradeoff in macaque area LIP , 2014, eLife.

[7]  Alexandre Pouget,et al.  Optimal multisensory decision-making in a reaction-time task , 2014, eLife.

[8]  P. Cisek,et al.  From anticipation to action, the role of dopamine in perceptual decision making: an fMRI-tyrosine depletion study. , 2012, Journal of neurophysiology.

[9]  Shih-Wei Wu,et al.  Dynamic combination of sensory and reward information under time pressure , 2018, bioRxiv.

[10]  James L. McClelland,et al.  The time course of perceptual choice: the leaky, competing accumulator model. , 2001, Psychological review.

[11]  Zachary P. Kilpatrick,et al.  Stochastic models of evidence accumulation in changing environments , 2015, bioRxiv.

[12]  Jochen Ditterich Stochastic models of decisions about motion direction: behavior and physiology , 2006 .

[13]  C. H. Donahue,et al.  Metaplasticity as a Neural Substrate for Adaptive Learning and Choice under Uncertainty , 2017, Neuron.

[14]  Roger Ratcliff,et al.  A Theory of Memory Retrieval. , 1978 .

[15]  Peter R Murphy,et al.  Global gain modulation generates time-dependent urgency during perceptual choice in humans , 2016, Nature Communications.

[16]  Francis Tuerlinckx,et al.  Do the Dynamics of Prior Information Depend on Task Context? An Analysis of Optimal Performance and an Empirical Test , 2012, Front. Psychology.

[17]  Scott D. Brown,et al.  Revisiting the Evidence for Collapsing Boundaries and Urgency Signals in Perceptual Decision-Making , 2015, The Journal of Neuroscience.

[18]  John D Murray,et al.  A flexible framework for simulating and fitting generalized drift-diffusion models , 2020, eLife.

[19]  Philip Holmes,et al.  Simple Neural Networks that Optimize Decisions , 2005, Int. J. Bifurc. Chaos.

[20]  Roger Ratcliff,et al.  The Diffusion Decision Model: Theory and Data for Two-Choice Decision Tasks , 2008, Neural Computation.

[21]  Bingni W. Brunton,et al.  Distinct effects of prefrontal and parietal cortex inactivations on an accumulation of evidence task in the rat , 2015, bioRxiv.

[22]  Timothy D. Hanks,et al.  Bounded Integration in Parietal Cortex Underlies Decisions Even When Viewing Duration Is Dictated by the Environment , 2008, The Journal of Neuroscience.

[23]  R. Ratcliff Theoretical interpretations of the speed and accuracy of positive and negative responses. , 1985, Psychological review.

[24]  Roger Ratcliff,et al.  Comparing fixed and collapsing boundary versions of the diffusion model. , 2016, Journal of mathematical psychology.

[25]  Jonathan D. Cohen,et al.  Reward rate optimization in two-alternative decision making: empirical tests of theoretical predictions. , 2009, Journal of experimental psychology. Human perception and performance.

[26]  R. Ratcliff,et al.  Bias in the Brain: A Diffusion Model Analysis of Prior Probability and Potential Payoff , 2012, The Journal of Neuroscience.

[27]  R. Carpenter,et al.  The influence of urgency on decision time , 2000, Nature Neuroscience.

[28]  P. Cisek,et al.  Decisions in Changing Conditions: The Urgency-Gating Model , 2009, The Journal of Neuroscience.

[29]  S. Piantadosi One parameter is always enough , 2018, AIP Advances.

[30]  Marius Usher,et al.  The Timescale of Perceptual Evidence Integration Can Be Adapted to the Environment , 2013, Current Biology.

[31]  John D. Murray,et al.  Effects of Altered Excitation-Inhibition Balance on Decision Making in a Cortical Circuit Model , 2017, The Journal of Neuroscience.

[32]  Konstantinos Tsetsos,et al.  Action Planning and the Timescale of Evidence Accumulation , 2015, PloS one.

[33]  Tali Sharot,et al.  Evidence accumulation is biased by motivation: A computational account , 2019, PLoS Comput. Biol..

[34]  A. Pouget,et al.  The Cost of Accumulating Evidence in Perceptual Decision Making , 2012, The Journal of Neuroscience.

[35]  K. H. Britten,et al.  Neuronal correlates of a perceptual decision , 1989, Nature.

[36]  Philip Holmes,et al.  Can Monkeys Choose Optimally When Faced with Noisy Stimuli and Unequal Rewards? , 2009, PLoS Comput. Biol..

[37]  Aaron J Levi,et al.  Strategic and Dynamic Temporal Weighting for Perceptual Decisions in Humans and Macaques , 2018, eNeuro.

[38]  P. Cisek,et al.  Deliberation and Commitment in the Premotor and Primary Motor Cortex during Dynamic Decision Making , 2014, Neuron.

[39]  D Laming,et al.  Autocorrelation of choice-reaction times. , 1979, Acta psychologica.

[40]  Gaurav Malhotra,et al.  Time-varying decision boundaries: insights from optimality analysis , 2017, Psychonomic bulletin & review.

[41]  James L. McClelland,et al.  Integration of Sensory and Reward Information during Perceptual Decision-Making in Lateral Intraparietal Cortex (LIP) of the Macaque Monkey , 2010, PloS one.

[42]  Long Ding,et al.  Ongoing, rational calibration of reward-driven perceptual biases , 2018, bioRxiv.

[43]  Christopher R Fetsch,et al.  The influence of evidence volatility on choice, reaction time and confidence in a perceptual decision , 2016, eLife.

[44]  R. Ratcliff,et al.  Estimating parameters of the diffusion model: Approaches to dealing with contaminant reaction times and parameter variability , 2002, Psychonomic bulletin & review.

[45]  Timothy D. Hanks,et al.  Causal contribution and dynamical encoding in the striatum during evidence accumulation , 2018, bioRxiv.

[46]  P. Holmes,et al.  MODELING A SIMPLE CHOICE TASK: STOCHASTIC DYNAMICS OF MUTUALLY INHIBITORY NEURAL GROUPS , 2001 .

[47]  Scott D. Brown,et al.  Diffusion Decision Model: Current Issues and History , 2016, Trends in Cognitive Sciences.

[48]  J. Ditterich Evidence for time‐variant decision making , 2006, The European journal of neuroscience.

[49]  A. Voss,et al.  Interpreting the parameters of the diffusion model: An empirical validation , 2004, Memory & cognition.

[50]  H. Woodrow The measurement of attention , 1914 .

[51]  Jonathan D. Cohen,et al.  The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced-choice tasks. , 2006, Psychological review.

[52]  Timothy D. Hanks,et al.  Elapsed Decision Time Affects the Weighting of Prior Probability in a Perceptual Decision Task , 2011, The Journal of Neuroscience.

[53]  I. J. Myung,et al.  Counting probability distributions: Differential geometry and model selection , 2000, Proc. Natl. Acad. Sci. USA.

[54]  A. Diederich,et al.  Modeling the effects of payoff on response bias in a perceptual discrimination task: Bound-change, drift-rate-change, or two-stage-processing hypothesis , 2006, Perception & psychophysics.

[55]  Simon P Kelly,et al.  The role of premature evidence accumulation in making difficult perceptual decisions under temporal uncertainty , 2019, eLife.

[56]  Rainer Storn,et al.  Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces , 1997, J. Glob. Optim..

[57]  J. Gold,et al.  The caudate nucleus contributes causally to decisions that balance reward and uncertain visual information , 2020, eLife.

[58]  Samuel W Cheadle,et al.  Adaptive Gain Control during Human Perceptual Choice , 2014, Neuron.

[59]  D. Rosenbaum,et al.  Timing of behavior : neural, psychological, and computational perspectives , 1998 .

[60]  Scott D. Brown,et al.  Cortico-striatal connections predict control over speed and accuracy in perceptual decision making , 2010, Proceedings of the National Academy of Sciences.

[61]  D A Rosenbaum,et al.  Timing and reaction time. , 2001, Journal of experimental psychology. General.

[62]  Paul Cisek,et al.  Decision making by urgency gating: theory and experimental support. , 2012, Journal of neurophysiology.

[63]  James L. McClelland,et al.  Payoff Information Biases a Fast Guess Process in Perceptual Decision Making under Deadline Pressure: Evidence from Behavior, Evoked Potentials, and Quantitative Model Comparison , 2015, The Journal of Neuroscience.

[64]  Carol A. Seger,et al.  Categorical evidence, confidence, and urgency during probabilistic categorization , 2016, NeuroImage.

[65]  James L. McClelland,et al.  Dynamic Integration of Reward and Stimulus Information in Perceptual Decision-Making , 2011, PloS one.

[66]  J. Doye,et al.  Global Optimization by Basin-Hopping and the Lowest Energy Structures of Lennard-Jones Clusters Containing up to 110 Atoms , 1997, cond-mat/9803344.

[67]  Bingni W. Brunton,et al.  Distinct relationships of parietal and prefrontal cortices to evidence accumulation , 2014, Nature.

[68]  Lixing Han,et al.  Implementing the Nelder-Mead simplex algorithm with adaptive parameters , 2010, Computational Optimization and Applications.

[69]  M. Shadlen,et al.  Response of Neurons in the Lateral Intraparietal Area during a Combined Visual Discrimination Reaction Time Task , 2002, The Journal of Neuroscience.

[70]  J. Gold,et al.  Neural computations that underlie decisions about sensory stimuli , 2001, Trends in Cognitive Sciences.

[71]  W. Edwards Optimal strategies for seeking information: Models for statistics, choice reaction times, and human information processing ☆ , 1965 .

[72]  Joel Nothman,et al.  SciPy 1.0-Fundamental Algorithms for Scientific Computing in Python , 2019, ArXiv.

[73]  Marius Usher,et al.  Non-monotonic Temporal-Weighting Indicates a Dynamically Modulated Evidence-Integration Mechanism , 2016, PLoS Comput. Biol..

[74]  Birte U. Forstmann,et al.  Striatal activation reflects urgency in perceptual decision making , 2016, NeuroImage.

[75]  C. Summerfield,et al.  Economic Value Biases Uncertain Perceptual Choices in the Parietal and Prefrontal Cortices , 2010, Front. Hum. Neurosci..

[76]  F. Ashby A biased random walk model for two choice reaction times , 1983 .

[77]  Tobias Teichert,et al.  The importance of decision onset. , 2016, Journal of neurophysiology.

[78]  A. Blangero,et al.  Dynamic Interplay of Value and Sensory Information in High-Speed Decision Making , 2018, Current Biology.

[79]  Timothy E. J. Behrens,et al.  Learning the value of information in an uncertain world , 2007, Nature Neuroscience.

[80]  M. Shadlen,et al.  Decision-making with multiple alternatives , 2008, Nature Neuroscience.

[81]  Naoshige Uchida,et al.  The Limits of Deliberation in a Perceptual Decision Task , 2013, Neuron.

[82]  O. Hikosaka,et al.  A neural correlate of response bias in monkey caudate nucleus , 2002, Nature.