Optimal models of decision-making in dynamic environments

Nature is in constant flux, so animals must account for changes in their environment when making decisions. How animals learn the timescale of such changes and adapt their decision strategies accordingly is not well understood. Recent psychophysical experiments have shown humans and other animals can achieve near-optimal performance at two alternative forced choice (2AFC) tasks in dynamically changing environments. Characterization of performance requires the derivation and analysis of computational models of optimal decision-making policies on such tasks. We review recent theoretical work in this area, and discuss how models compare with subjects' behavior in tasks where the correct choice or evidence quality changes in dynamic, but predictable, ways.

[1]  D. Knill,et al.  The Bayesian brain: the role of uncertainty in neural coding and computation , 2004, Trends in Neurosciences.

[2]  Vijay Balasubramanian,et al.  A bias–variance trade-off governs individual differences in on-line learning in an unpredictable environment , 2018, Nature Human Behaviour.

[3]  Christopher C. Pack,et al.  Bidirectional manipulation of GABAergic inhibition in MT: A comparison of neuronal and psychophysical performance , 2014 .

[4]  Jonathan D. Cohen,et al.  The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced-choice tasks. , 2006, Psychological review.

[5]  A. Pouget,et al.  Not Noisy, Just Wrong: The Role of Suboptimal Inference in Behavioral Variability , 2012, Neuron.

[6]  P. Cisek,et al.  The Basal Ganglia Do Not Select Reach Targets but Control the Urgency of Commitment , 2017, Neuron.

[7]  Anne E. Urai,et al.  Pupil-linked arousal is driven by decision uncertainty and alters serial choice bias , 2017, Nature Communications.

[8]  J. Gold,et al.  Arousal-related adjustments of perceptual biases optimize perception in dynamic environments , 2017, Nature Human Behaviour.

[9]  Brandon M. Turner,et al.  Some task demands induce collapsing bounds: Evidence from a behavioral analysis , 2018, Psychonomic Bulletin & Review.

[10]  N. Anderson.  Effect of first-order conditional probability in two-choice learning situation. , 1960, Journal of experimental psychology.

[11]  Marius Usher,et al.  The Timescale of Perceptual Evidence Integration Can Be Adapted to the Environment , 2013, Current Biology.

[12]  J. Gold,et al.  Coupled Decision Processes Update and Maintain Saccadic Priors in a Dynamic Environment , 2017, The Journal of Neuroscience.

[13]  Joshua I. Gold,et al.  Bayesian Online Learning of the Hazard Rate in Change-Point Problems , 2010, Neural Computation.

[14]  Bingni W. Brunton,et al.  Rats and Humans Can Optimally Accumulate Evidence for Decision-Making , 2013, Science.

[15]  C. White,et al.  Decomposing bias in different types of simple decisions. , 2014, Journal of experimental psychology. Learning, memory, and cognition.

[16]  J. Movshon,et al.  The analysis of visual motion: a comparison of neuronal and psychophysical performance , 1992, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[17]  Naomi Ehrich Leonard,et al.  Can Post-Error Dynamics Explain Sequential Reaction Time Patterns? , 2012, Front. Psychology.

[18]  Surya Ganguli,et al.  A theory of multineuronal dimensionality, dynamics and measurement , 2017, bioRxiv.

[19]  Joseph W Kable,et al.  Normative evidence accumulation in unpredictable environments , 2015, eLife.

[20]  Anne E. Urai,et al.  Choice History Biases Subsequent Evidence Accumulation , 2018, bioRxiv.

[21]  Vijay Balasubramanian,et al.  On the Complexity of Prediction Strategies in Noisy and Changing Environments , 2018 .

[22]  Veliz-CubaAlan,et al.  Evidence accumulation and change rate inference in dynamic environments , 2017 .

[23]  Alexandre Pouget,et al.  Optimal policy for value-based decision-making , 2016, Nature Communications.

[24]  W. Geisler.  Ideal Observer Analysis , 2002 .

[25]  Braden A. Purcell,et al.  Hierarchical decision processes that operate over distinct timescales underlie choice and changes in strategy , 2016, Proceedings of the National Academy of Sciences.

[26]  A. Diederich,et al.  Modeling the effects of payoff on response bias in a perceptual discrimination task: Bound-change, drift-rate-change, or two-stage-processing hypothesis , 2006, Perception & psychophysics.

[27]  Alexandre Hyafil,et al.  Response outcomes gate the impact of expectations on perceptual decisions , 2018, Nature Communications.

[28]  Joseph F. Hanna,et al.  The structure of responses to a sequence of binary events , 1966 .

[29]  J. Macke,et al.  Quantifying the effect of intertrial dependence on perceptual decisions. , 2014, Journal of vision.

[30]  Marius Usher,et al.  Decisions reduce sensitivity to subsequent information , 2015, Proceedings of the Royal Society B: Biological Sciences.

[31]  Zachary P. Kilpatrick,et al.  Evidence accumulation and change rate inference in dynamic environments , 2016, bioRxiv.

[32]  Jennifer S Trueblood,et al.  Bayesian analysis of the piecewise diffusion decision model , 2018, Behavior research methods.

[33]  A. Pouget,et al.  The Cost of Accumulating Evidence in Perceptual Decision Making , 2012, The Journal of Neuroscience.

[34]  Gaurav Malhotra,et al.  Overcoming Indecision by Changing the Decision Boundary , 2017, Journal of experimental psychology. General.

[35]  Anne E. Urai,et al.  Confirmation Bias through Selective Overweighting of Choice-Consistent Evidence , 2018, Current Biology.

[36]  Bridgette Johnson,et al.  Characterization of decision commitment rule alterations during an auditory change detection task. , 2017, Journal of neurophysiology.

[37]  Sophie Deneve,et al.  Making Decisions with Unknown Sensory Reliability , 2012, Front. Neurosci..

[38]  Michael D. Lee,et al.  Time-varying boundaries for diffusion models of decision making and response time , 2014, Front. Psychol..

[39]  R. Ratcliff.  Theoretical interpretations of the speed and accuracy of positive and negative responses. , 1985, Psychological review.

[40]  Jonathan D. Cohen,et al.  Sequential effects: Superstition or rational behavior? , 2008, NIPS.

[41]  Charles D. Kopec,et al.  Posterior parietal cortex represents sensory history and mediates its effects on behaviour , 2017, Nature.

[42]  Scott D. Brown,et al.  The computations that support simple decision-making: A comparison between the diffusion and urgency-gating models , 2017, Scientific Reports.

[43]  J. Gold,et al.  The neural basis of decision making. , 2007, Annual review of neuroscience.

[44]  Jerome R. Busemeyer,et al.  Learning to allocate limited time to decisions with different expected outcomes , 2016, Cognitive Psychology.

[45]  Reza Ebrahimpour,et al.  Residual Information of Previous Decision Affects Evidence Accumulation in Current Decision , 2016, Front. Behav. Neurosci..

[46]  Scott D. Brown,et al.  Revisiting the Evidence for Collapsing Boundaries and Urgency Signals in Perceptual Decision-Making , 2015, The Journal of Neuroscience.

[47]  Kresimir Josic,et al.  Optimizing sequential decisions in the drift-diffusion model , 2018, bioRxiv.

[48]  Ryan P. Adams,et al.  Bayesian Online Changepoint Detection , 2007, 0710.3742.

[49]  Alexandre Pouget,et al.  Tuning the speed-accuracy trade-off to maximize reward rate in multisensory decision-making , 2015, eLife.

[50]  Andrew Heathcote,et al.  A new framework for modeling decisions about changing information: The Piecewise Linear Ballistic Accumulator model , 2016, Cognitive Psychology.

[51]  Roozbeh Kiani,et al.  Neural Mechanisms of Post-error Adjustments of Decision Policy in Parietal Cortex , 2016, Neuron.

[52]  Ahmed El Hady,et al.  Rats adopt the optimal timescale for evidence integration in a dynamic environment , 2018, Nature Communications.

[53]  Tobias H. Donner,et al.  Adaptive History Biases Result from Confidence-Weighted Accumulation of past Choices , 2017, The Journal of Neuroscience.

[54]  Paul Cisek,et al.  Decision making by urgency gating: theory and experimental support. , 2012, Journal of neurophysiology.

[55]  Paul Schrater,et al.  Inverse POMDP: Inferring What You Think from What You Do , 2018, ArXiv.

[56]  Scott D. Brown,et al.  Discriminating evidence accumulation from urgency signals in speeded decision making. , 2015, Journal of neurophysiology.

[57]  Roozbeh Kiani,et al.  A neural mechanism of speed-accuracy tradeoff in macaque area LIP , 2014, eLife.

[58]  S. Fernberger.  Interdependence of judgments within the series for the method of constant stimuli. , 1920 .

[59]  A. U.S.,et al.  Predictability , Complexity , and Learning , 2002 .

[60]  Robert C. Wilson,et al.  Rational regulation of learning dynamics by pupil–linked arousal systems , 2012, Nature Neuroscience.

[61]  Zachary P. Kilpatrick,et al.  Stochastic models of evidence accumulation in changing environments , 2015, bioRxiv.