Ongoing, rational calibration of reward-driven perceptual biases

Decision-making is often interpreted in terms of normative computations that maximize a particular reward function for stable, average behaviors. Aberrations from the reward-maximizing solutions, either across subjects or across different sessions for the same subject, are often interpreted as reflecting poor learning or physical limitations. Here we show that such aberrations may instead reflect the involvement of additional satisficing and heuristic principles. For an asymmetric-reward perceptual decision-making task, three monkeys produced adaptive biases in response to changes in reward asymmetries and perceptual sensitivity. Their choices and response times were consistent with a normative accumulate-to-bound process. However, their context-dependent adjustments to this process deviated slightly but systematically from the reward-maximizing solutions. These adjustments were instead consistent with a rational process to find satisficing solutions based on the gradient of each monkey’s reward-rate function. These results suggest new dimensions for assessing the rational and idiosyncratic aspects of flexible decision-making.

[1]  Roger Ratcliff,et al.  A Theory of Memory Retrieval. , 1978 .

[2]  N. Uchida,et al.  The dorsomedial striatum encodes net expected return, critical for energizing performance vigor , 2013, Nature Neuroscience.

[3]  G. A. Barnard,et al.  Sequential Tests in Industrial Statistics , 1946 .

[4]  W. Edwards Optimal strategies for seeking information: Models for statistics, choice reaction times, and human information processing ☆ , 1965 .

[5]  Jonathan D. Cohen,et al.  The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced-choice tasks. , 2006, Psychological review.

[6]  Naomi Ehrich Leonard,et al.  A comparative study of drift diffusion and linear ballistic accumulator models in a reward maximization perceptual choice task , 2014, Front. Neurosci..

[7]  R. Nosofsky An Exemplar-Based Random-Walk Model of Speeded Categorization and Absolute Judgment , 2019, Choice, Decision, and Measurement: Essays in Honor of R. Duncan Luce.

[8]  M. Philiastides,et al.  Temporal Characteristics of the Influence of Punishment on Perceptual Decision Making in the Human Brain , 2013, The Journal of Neuroscience.

[9]  R. Ratcliff,et al.  What cognitive processes drive response biases? A diffusion model analysis , 2011, Judgment and Decision Making.

[10]  Philip L. Smith,et al.  Psychology and neurobiology of simple decisions , 2004, Trends in Neurosciences.

[11]  Thomas V. Wiecki,et al.  fMRI and EEG Predictors of Dynamic Decision Parameters during Human Reinforcement Learning , 2015, The Journal of Neuroscience.

[12]  Ian Krajbich,et al.  Visual fixations and the computation and comparison of value in simple choice , 2010, Nature Neuroscience.

[13]  Timothy D. Hanks,et al.  Elapsed Decision Time Affects the Weighting of Prior Probability in a Perceptual Decision Task , 2011, The Journal of Neuroscience.

[14]  A. Pouget,et al.  Variance as a Signature of Neural Computations during Decision Making , 2011, Neuron.

[15]  Gerd Gigerenzer,et al.  Heuristic decision making. , 2011, Annual review of psychology.

[16]  M. Shadlen,et al.  The effect of stimulus strength on the speed and accuracy of a perceptual decision. , 2005, Journal of vision.

[17]  Takeo Watanabe,et al.  Temporally Extended Dopamine Responses to Perceptually Demanding Reward-Predictive Stimuli , 2010, The Journal of Neuroscience.

[18]  Wynn C. Stirling Satisficing Games and Decision Making: With Applications to Engineering and Computer Science , 2003 .

[19]  Razvan Pascanu,et al.  Overcoming catastrophic forgetting in neural networks , 2016, Proceedings of the National Academy of Sciences.

[20]  Xiao-Jing Wang,et al.  Speed-accuracy tradeoff by a control signal with balanced excitation and inhibition. , 2015, Journal of neurophysiology.

[21]  Jonathan W. Pillow,et al.  Single-trial spike trains in parietal cortex reveal discrete steps during decision-making , 2015, Science.

[22]  J. Gold,et al.  Distinct Representations of a Perceptual Decision and the Associated Oculomotor Plan in the Monkey Lateral Intraparietal Area , 2011, The Journal of Neuroscience.

[23]  Christof Koch,et al.  The drift diffusion model can account for value-based choice response times under high and low time pressure , 2010 .

[24]  Christopher R Fetsch,et al.  The influence of evidence volatility on choice, reaction time and confidence in a perceptual decision , 2016, eLife.

[25]  M. Shadlen,et al.  Response of Neurons in the Lateral Intraparietal Area during a Combined Visual Discrimination Reaction Time Task , 2002, The Journal of Neuroscience.

[26]  R. Ratcliff,et al.  Estimating parameters of the diffusion model: Approaches to dealing with contaminant reaction times and parameter variability , 2002, Psychonomic bulletin & review.

[27]  Xiao-Jing Wang,et al.  Cortico–basal ganglia circuit mechanism for a decision threshold in reaction time tasks , 2006, Nature Neuroscience.

[28]  W T Newsome,et al.  Target selection for saccadic eye movements: direction-selective visual responses in the superior colliculus. , 2001, Journal of neurophysiology.

[29]  J. Gold,et al.  Separate, Causal Roles of the Caudate in Saccadic Choice and Execution in a Perceptual Decision Task , 2012, Neuron.

[30]  L. Ding Distinct dynamics of ramping activity in the frontal cortex and caudate nucleus in monkeys. , 2015, Journal of neurophysiology.

[31]  A. Voss,et al.  Interpreting the parameters of the diffusion model: An empirical validation , 2004, Memory & cognition.

[32]  R. Ratcliff,et al.  A comparison of macaque behavior and superior colliculus neuronal activity to predictions from models of two-choice decisions. , 2003, Journal of neurophysiology.

[33]  L. Stone,et al.  Effects of Prior Information and Reward on Oculomotor and Perceptual Choices , 2008, The Journal of Neuroscience.

[34]  Daniel E. Shub,et al.  The Role of Response Bias in Perceptual Learning , 2015, Journal of experimental psychology. Learning, memory, and cognition.

[35]  J. Gold,et al.  Neural correlates of perceptual decision making before, during, and after decision commitment in monkey frontal eye field. , 2012, Cerebral cortex.

[36]  Roger Ratcliff,et al.  Age-related differences in diffusion model boundary optimality with both trial-limited and time-limited tasks , 2011, Psychonomic Bulletin & Review.

[37]  James L. McClelland,et al.  Integration of Sensory and Reward Information during Perceptual Decision-Making in Lateral Intraparietal Cortex (LIP) of the Macaque Monkey , 2010, PloS one.

[38]  Xiao-Jing Wang,et al.  Role of the Indirect Pathway of the Basal Ganglia in Perceptual Decision Making , 2015, The Journal of Neuroscience.

[39]  R. Nosofsky,et al.  An exemplar-based random walk model of speeded classification. , 1997, Psychological review.

[40]  R. Ratcliff,et al.  Connectionist and diffusion models of reaction time. , 1999, Psychological review.

[41]  R. Ratcliff,et al.  The effects of aging on the speed-accuracy compromise: Boundary optimality in the diffusion model. , 2010, Psychology and aging.

[42]  Tobias Teichert,et al.  Suboptimal Integration of Reward Magnitude and Prior Reward Likelihood in Categorical Decisions by Monkeys , 2010, Front. Neurosci..

[43]  K. H. Britten,et al.  Neuronal correlates of a perceptual decision , 1989, Nature.

[44]  J. Wolfowitz,et al.  Optimum Character of the Sequential Probability Ratio Test , 1948 .

[45]  Francis Tuerlinckx,et al.  Fitting the ratcliff diffusion model to experimental data , 2007, Psychonomic bulletin & review.

[46]  H. Simon,et al.  Rational choice and the structure of the environment. , 1956, Psychological review.

[47]  C. Law,et al.  The relative influences of priors and sensory evidence on an oculomotor decision variable during perceptual learning. , 2008, Journal of neurophysiology.

[48]  J. Gold,et al.  The neural basis of decision making. , 2007, Annual review of neuroscience.

[49]  Gerd Gigerenzer,et al.  Moral Satisficing: Rethinking Moral Behavior as Bounded Rationality , 2010, Top. Cogn. Sci..

[50]  Bruce G Cumming,et al.  Reward modulates the effect of visual cortical microstimulation on perceptual decisions , 2015, eLife.

[51]  James L. McClelland,et al.  Dynamic Integration of Reward and Stimulus Information in Perceptual Decision-Making , 2011, PloS one.

[52]  P. Matthews,et al.  Independent anatomical and functional measures of the V1/V2 boundary in human visual cortex. , 2005, Journal of vision.

[53]  J. Andel Sequential Analysis , 2022, The SAGE Encyclopedia of Research Design.

[54]  Philip Holmes,et al.  Can Monkeys Choose Optimally When Faced with Noisy Stimuli and Unequal Rewards? , 2009, PLoS Comput. Biol..

[55]  John T Serences,et al.  Value-Based Modulations in Human Visual Cortex , 2008, Neuron.

[56]  Shlomo Zilberstein,et al.  Models of Bounded Rationality , 1995 .

[57]  Sergei Gepshtein,et al.  Intermittent regime of brain activity at the early, bias-guided stage of perceptual learning. , 2016, Journal of vision.

[58]  A. Wierzbicki A Mathematical Basis for Satisficing Decision Making , 1982 .

[59]  Masatoshi Sakawa,et al.  An interactive fuzzy satisficing method for multiobjective nonconvex programming problems with fuzzy numbers through coevolutionary genetic algorithms , 2001, IEEE Trans. Syst. Man Cybern. Part B.

[60]  Paul Cisek,et al.  Decision making by urgency gating: theory and experimental support. , 2012, Journal of neurophysiology.

[61]  M. Shadlen,et al.  Decision-making with multiple alternatives , 2008, Nature Neuroscience.

[62]  A. Pouget,et al.  The Cost of Accumulating Evidence in Perceptual Decision Making , 2012, The Journal of Neuroscience.

[63]  Karl R. Gegenfurtner,et al.  Interaction of motion and color in the visual pathways , 1996, Trends in Neurosciences.

[64]  E. Wagenmakers,et al.  Psychological interpretation of the ex-Gaussian and shifted Wald parameters: A diffusion model analysis , 2009, Psychonomic bulletin & review.

[65]  Marek S. Wartak,et al.  Drift–Diffusion Model , 2019, Field Guide to Solid State Physics.

[66]  Masatoshi Sakawa,et al.  An Interactive Fuzzy Satisficing Method for Multiobjective Linear Fractional Programs with Block Angular Structure , 1997, Cybern. Syst..

[67]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[68]  Philip L. Smith,et al.  A comparison of sequential sampling models for two-choice reaction time. , 2004, Psychological review.

[69]  C. White,et al.  Decomposing bias in different types of simple decisions. , 2014, Journal of experimental psychology. Learning, memory, and cognition.

[70]  Rajesh P. N. Rao,et al.  Decision Making Under Uncertainty: A Neural Model Based on Partially Observable Markov Decision Processes , 2010, Front. Comput. Neurosci..

[71]  J. Gold,et al.  Caudate Encodes Multiple Computations for Perceptual Decisions , 2010, The Journal of Neuroscience.

[72]  M. Shadlen,et al.  Decision Making and Sequential Sampling from Memory , 2016, Neuron.

[73]  H. Simon,et al.  Theories of Decision-Making in Economics and Behavioural Science , 1966 .

[74]  R. Ratcliff,et al.  Bias in the Brain: A Diffusion Model Analysis of Prior Probability and Potential Payoff , 2012, The Journal of Neuroscience.

[75]  M Zacksenhouse,et al.  Robust versus optimal strategies for two-alternative forced choice tasks. , 2010, Journal of mathematical psychology.

[76]  W. Newsome,et al.  Neural basis of a perceptual decision in the parietal cortex (area LIP) of the rhesus monkey. , 2001, Journal of neurophysiology.

[77]  R. Ratcliff Theoretical interpretations of the speed and accuracy of positive and negative responses. , 1985, Psychological review.

[78]  G Gigerenzer,et al.  Reasoning the fast and frugal way: models of bounded rationality. , 1996, Psychological review.

[79]  Thomas V. Wiecki,et al.  HDDM: Hierarchical Bayesian estimation of the Drift-Diffusion Model in Python , 2013, Front. Neuroinform..

[80]  M. Goldberg,et al.  Memory-Guided Saccade to a Distractor Flashed During the Delay Period of a Response of Neurons in the Lateral Intraparietal Area , 2015 .

[81]  A. Pouget,et al.  Probabilistic Synapses , 2014, 1410.1029.

[82]  M. Shadlen,et al.  Representation of Confidence Associated with a Decision by Neurons in the Parietal Cortex , 2009, Science.

[83]  Joshua I Gold,et al.  Correlates of Perceptual Learning in an Oculomotor Decision Variable , 2009, The Journal of Neuroscience.

[84]  A. Diederich,et al.  Modeling the effects of payoff on response bias in a perceptual discrimination task: Bound-change, drift-rate-change, or two-stage-processing hypothesis , 2006, Perception & psychophysics.

[85]  Gerd Gigerenzer,et al.  Models of ecological rationality: the recognition heuristic. , 2002, Psychological review.

[86]  Jonathan D. Cohen,et al.  Reward rate optimization in two-alternative decision making: empirical tests of theoretical predictions. , 2009, Journal of experimental psychology. Human perception and performance.

[87]  J. Gold,et al.  Banburismus and the Brain Decoding the Relationship between Sensory Stimuli, Decisions, and Reward , 2002, Neuron.

[88]  Timothy D. Hanks,et al.  Microstimulation of macaque area LIP affects decision-making in a motion discrimination task , 2006, Nature Neuroscience.

[89]  Corey J. Bohil,et al.  Base-rate and payoff effects in multidimensional perceptual categorization. , 1998, Journal of Experimental Psychology. Learning, Memory and Cognition.

[90]  Christof Koch,et al.  The Drift Diffusion Model Can Account for the Accuracy and Reaction Time of Value-Based Choices Under High and Low Time Pressure , 2010, Judgment and Decision Making.

[91]  Wynn C. Stirling,et al.  A theory of satisficing decisions and control , 1998, IEEE Trans. Syst. Man Cybern. Part A.

[92]  S. Klein,et al.  Measuring, estimating, and understanding the psychometric function: A commentary , 2001, Perception & psychophysics.

[93]  C. Summerfield,et al.  Economic Value Biases Uncertain Perceptual Choices in the Parietal and Prefrontal Cortices , 2010, Front. Hum. Neurosci..

[94]  F. Ashby A biased random walk model for two choice reaction times , 1983 .

[95]  M. Sommer,et al.  Satisficing in split-second decision making is characterized by strategic cue discounting. , 2016, Journal of experimental psychology. Learning, memory, and cognition.