A theoretical account of cognitive effects in delay discounting

Although delay discounting, the attenuation of the value of future rewards, is a robust finding, the mechanism of discounting is not known. We propose a potential mechanism for delay discounting such that discounting emerges from a search process that is trying to determine what rewards will be available in the future. In this theory, the delay dependence of the discounting of future expected rewards arises from three assumptions. First, that the evaluation of outcomes involves a search process. Second, that the value is assigned to an outcome proportionally to how easy it is to find. Third, that outcomes that are less delayed are typically easier for the search process to find. By relaxing this third assumption (e.g. by assuming that episodically‐cued outcomes are easier to find), our model suggests that it is possible to dissociate discounting from delay. Our theory thereby explains the empirical result that discounting is slower to episodically‐imagined outcomes, because these outcomes are easier for the search process to find. Additionally, the theory explains why improving cognitive resources such as working memory slows discounting, by improving searches and thereby making rewards easier to find. The three assumptions outlined here are likely to be instantiated during deliberative decision‐making, but are unlikely in habitual decision‐making. We model two simple implementations of this theory and show that they unify empirical results about the role of cognitive function in delay discounting, and make new neural, behavioral, and pharmacological predictions.

[1]  P. Samuelson A Note on Measurement of Utility , 1937 .

[2]  E. Tolman Prediction of vicarious trial and error by means of the schematic sowbug. , 1939 .

[3]  F. Restle Discrimination of cues in mazes: a resolution of the place-vs.-response question. , 1957, Psychological review.

[4]  W. Mischel,et al.  Waiting for rewards and punishments: effects of time and probability on choice. , 1967, Journal of personality and social psychology.

[5]  J. Cowan,et al.  Excitatory and inhibitory interactions in localized populations of model neurons. , 1972, Biophysical journal.

[6]  Allen Newell,et al.  Human Problem Solving. , 1973 .

[7]  E. Loftus,et al.  Reconstruction of automobile destruction: An example of the interaction between language and memory , 1974 .

[8]  W. Mischel,et al.  Instrumental ideation in delay of gratification. , 1974, Child development.

[9]  Nils J. Nilsson,et al.  Artificial Intelligence , 1974, IFIP Congress.

[10]  A. Tversky,et al.  Prospect theory: analysis of decision under risk , 1979 .

[11]  C. Barnes Memory deficits associated with senescence: a neurophysiological and behavioral study in the rat. , 1979, Journal of comparative and physiological psychology.

[12]  Teuvo Kohonen,et al.  Content-addressable memories , 1980 .

[13]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[14]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[15]  G. Ainslie Picoeconomics: The Strategic Interaction of Successive Motivational States within the Person , 1992 .

[16]  A. Tversky,et al.  Choice in Context: Tradeoff Contrast and Extremeness Aversion , 1992 .

[17]  R. Baumeister,et al.  Losing Control: How and Why People Fail at Self-Regulation , 1994 .

[18]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[19]  E. Rolls,et al.  The Orbitofrontal Cortex , 2019 .

[20]  J. D. McGaugh,et al.  Inactivation of Hippocampus or Caudate Nucleus with Lidocaine Differentially Affects Expression of Place and Response Learning , 1996, Neurobiology of Learning and Memory.

[21]  K. Zhang,et al.  Representation of spatial orientation by the intrinsic dynamics of the head-direction cell ensemble: a theory , 1996, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[22]  David I. Laibson,et al.  Golden Eggs and Hyperbolic Discounting , 1997 .

[23]  J. E. Mazur Choice, delay, probability, and conditioned reinforcement , 1997 .

[24]  Nils J. Nilsson,et al.  Artificial Intelligence: A New Synthesis , 1997 .

[25]  Peter D. Sozou,et al.  On hyperbolic discounting and uncertain hazard rates , 1998, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[26]  K. Nader,et al.  A Two-Separate-Motivational-Systems Hypothesis of Opioid Addiction , 1998, Pharmacology Biochemistry and Behavior.

[27]  A. Redish Beyond the Cognitive Map: From Place Cells to Episodic Memory , 1999 .

[28]  Kenji Doya,et al.  What are the computations of the cerebellum, the basal ganglia and the cerebral cortex? , 1999, Neural Networks.

[29]  H. Rachlin The Science of Self-Control , 2004 .

[30]  T. Sejnowski,et al.  Neurocomputational models of working memory , 2000, Nature Neuroscience.

[31]  Steven W Anderson,et al.  Decision-making deficits, linked to a dysfunctional ventromedial prefrontal cortex, revealed in alcohol and stimulant abusers , 2001, Neuropsychologia.

[32]  C. Atance,et al.  Episodic future thinking , 2001, Trends in Cognitive Sciences.

[33]  W. Bickel,et al.  Toward a behavioral economic understanding of drug dependence: delay discounting processes. , 2001, Addiction.

[34]  D. Wilkin,et al.  Neuron , 2001, Brain Research.

[35]  G. Ainslie Breakdown of will , 2001 .

[36]  J. Gold,et al.  Banburismus and the Brain Decoding the Relationship between Sensory Stimuli, Decisions, and Reward , 2002, Neuron.

[37]  Amy L Odum,et al.  Discounting of delayed health gains and losses by current, never- and ex-smokers of cigarettes. , 2002, Nicotine & tobacco research : official journal of the Society for Research on Nicotine and Tobacco.

[38]  M. El-Sabaawi Breakdown of Will , 2002 .

[39]  Roland E. Suri,et al.  TD models of reward predictive responses in dopamine neurons , 2002, Neural Networks.

[40]  Sridhar Mahadevan,et al.  Recent Advances in Hierarchical Reinforcement Learning , 2003, Discret. Event Dyn. Syst..

[41]  Yaacov Trope,et al.  Temporal construal. , 2003, Psychological review.

[42]  P. Whitney,et al.  Impulsive decision making and working memory. , 2003, Journal of experimental psychology. Learning, memory, and cognition.

[43]  N. Daw,et al.  Reinforcement learning models of the dopamine system and their behavioral implications , 2003 .

[44]  J. Seamans,et al.  The principal features and mechanisms of dopamine modulation in the prefrontal cortex , 2004, Progress in Neurobiology.

[45]  B. Knowlton,et al.  Contributions of striatal subregions to place and response learning. , 2004, Learning & memory.

[46]  Samuel M. McClure,et al.  Separate Neural Systems Value Immediate and Delayed Monetary Rewards , 2004, Science.

[47]  Gregory J. Madden,et al.  Impulsivity (delay discounting) as a predictor of acquisition of IV cocaine self-administration in female rats , 2005, Psychopharmacology.

[48]  Shoji Tanaka,et al.  Parametric study of dopaminergic neuromodulatory effects in a reduced model of the prefrontal cortex , 2005, Neurocomputing.

[49]  Marc W Howard,et al.  The temporal context model in spatial navigation and relational learning: toward a common explanation of medial temporal lobe function across domains. , 2005, Psychological review.

[50]  P. Dayan,et al.  Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control , 2005, Nature Neuroscience.

[51]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[52]  Y. Lacasse,et al.  From the authors , 2005, European Respiratory Journal.

[53]  C. Padoa-Schioppa,et al.  Neurons in the orbitofrontal cortex encode economic value , 2006, Nature.

[54]  S. Kapur,et al.  Dopamine, prediction error and associative learning: A model-based account , 2006, Network.

[55]  Warren K Bickel,et al.  Discounting of past outcomes. , 2006, Experimental and clinical psychopharmacology.

[56]  Kenji Doya,et al.  Humans Can Adopt Optimal Discounting Strategy under Real-Time Constraints , 2006, PLoS Comput. Biol..

[57]  H. Rachlin Notes on discounting. , 2006, Journal of the experimental analysis of behavior.

[58]  P. Dayan,et al.  Opinion TRENDS in Cognitive Sciences Vol.10 No.8 Full text provided by www.sciencedirect.com A normative perspective on motivation , 2022 .

[59]  Harold Pashler,et al.  Does working memory load lead to greater impulsivity? Commentary on Hinson, Jameson, and Whitney (2003). , 2006, Journal of experimental psychology. Learning, memory, and cognition.

[60]  A. Tversky,et al.  Prospect theory: an analysis of decision under risk — Source link , 2007 .

[61]  R. Baumeister,et al.  Self-control relies on glucose as a limited energy source: willpower is more than a metaphor. , 2007, Journal of personality and social psychology.

[62]  Timothy D. Wilson,et al.  Prospection: Experiencing the Future , 2007, Science.

[63]  Q. Huys Reinforcers and control : towards a computational aetiology of depression , 2007 .

[64]  Jane E. J. Ebert,et al.  The Fragility of Time: Time-Insensitivity and Valuation of the Near and Far Future , 2007, Manag. Sci..

[65]  Michael J. Frank,et al.  Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning , 2007, Proceedings of the National Academy of Sciences.

[66]  D. Hassabis,et al.  Patients with hippocampal amnesia cannot imagine new experiences , 2007, Proceedings of the National Academy of Sciences.

[67]  Jadin C. Jackson,et al.  Reconciling reinforcement learning models with behavioral extinction and renewal: implications for addiction, relapse, and problem gambling. , 2007, Psychological review.

[68]  Matthijs A. A. van der Meer,et al.  Integrating hippocampus and striatum in decision-making , 2007, Current Opinion in Neurobiology.

[69]  Daniel L. Schacter,et al.  Constructive memory: The ghosts of past and future , 2007, Nature.

[70]  H. Eichenbaum,et al.  The medial temporal lobe and recognition memory. , 2007, Annual review of neuroscience.

[71]  Adam Johnson,et al.  Neural Ensembles in CA3 Transiently Encode Paths Forward of the Animal at a Decision Point , 2007, The Journal of Neuroscience.

[72]  R. Buckner,et al.  Self-projection and the brain , 2007, Trends in Cognitive Sciences.

[73]  Alana T. Wong,et al.  Remembering the past and imagining the future: Common and distinct neural substrates during event construction and elaboration , 2007, Neuropsychologia.

[74]  D. Schacter,et al.  Remembering the past to imagine the future: the prospective brain , 2007, Nature Reviews Neuroscience.

[75]  Mark A. McDaniel,et al.  Prospective Memory : Cognitive, Neuroscience, Developmental, and Applied Perspectives , 2007 .

[76]  B. Balleine,et al.  Action Selection and Initiation in Instrumental Conditioning , 2007 .

[77]  Ronald J. Faber,et al.  Spent Resources: Self‐Regulatory Resource Availability Affects Impulse Buying , 2007 .

[78]  Matthew Botvinick,et al.  Goal-directed decision making in prefrontal cortex: a computational framework , 2008, NIPS.

[79]  Roger Ratcliff,et al.  The Diffusion Decision Model: Theory and Data for Two-Choice Decision Tasks , 2008, Neural Computation.

[80]  Warren K Bickel,et al.  Cigarette smokers discount past and future rewards symmetrically and more than controls: is discounting a measure of impulsivity? , 2008, Drug and alcohol dependence.

[81]  C. Hill The Rationality of Preference Construction (and the Irrationality of Rational Choice) , 2008 .

[82]  M. D’Esposito Working memory. , 2008, Handbook of clinical neurology.

[83]  Adam Johnson,et al.  Computing motivation: Incentive salience boosts of drug or appetite states , 2008, Behavioral and Brain Sciences.

[84]  Colin Camerer,et al.  Neuroeconomics: decision making and the brain , 2008 .

[85]  Aldo Rustichini,et al.  Cognitive skills affect economic preferences, strategic behavior, and job attachment , 2009, Proceedings of the National Academy of Sciences.

[86]  J. Anker,et al.  Impulsivity predicts the escalation of cocaine self-administration in rats , 2009, Pharmacology Biochemistry and Behavior.

[87]  Zeb Kurth-Nelson,et al.  Temporal-Difference Reinforcement Learning with Distributed Representations , 2009, PloS one.

[88]  Kyle S. Smith,et al.  Corticostriatal Interactions during Learning, Memory Processing, and Decision Making , 2009, The Journal of Neuroscience.

[89]  M. Roesch,et al.  Ventral Striatal Neurons Encode the Value of the Chosen Action in Rats Deciding between Differently Delayed or Sized Rewards , 2009, The Journal of Neuroscience.

[90]  Matthijs A. A. van der Meer,et al.  Frontiers in Integrative Neuroscience Integrative Neuroscience Covert Expectation-of-reward in Rat Ventral Striatum at Decision Points , 2022 .

[91]  M. Frank,et al.  Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation. , 2009, Nature neuroscience.

[92]  M. Roesch,et al.  A new perspective on the role of the orbitofrontal cortex in adaptive behaviour , 2009, Nature Reviews Neuroscience.

[93]  Ian Krajbich,et al.  Visual fixations and the computation and comparison of value in simple choice , 2010, Nature Neuroscience.

[94]  J. Lauwereyns The Anatomy of Bias: How Neural Circuits Weigh the Options , 2010 .

[95]  Joel Myerson,et al.  Pigeons' discounting of probabilistic and delayed reinforcers. , 2010, Journal of the experimental analysis of behavior.

[96]  R. Buckner The role of the hippocampus in prediction and imagination. , 2010, Annual review of psychology.

[97]  D. Durstewitz,et al.  Abrupt Transitions between Prefrontal Neural Ensemble States Accompany Behavioral Transitions during Rule Learning , 2010, Neuron.

[98]  Stefan M. Gruenert,et al.  Addiction: A Disorder of Choice , 2010 .

[99]  Jan Peters,et al.  Episodic Future Thinking Reduces Reward Delay Discounting through an Enhancement of Prefrontal-Mediotemporal Interactions , 2010, Neuron.

[100]  Zeb Kurth-Nelson,et al.  Neural models of delay discounting. , 2010 .

[101]  Matthijs A. A. van der Meer,et al.  Expectancies in Decision Making, Reinforcement Learning, and Ventral Striatum , 2009, Frontiers in neuroscience.

[102]  G. Madden,et al.  Impulsivity: The Behavioral and Neurological Science of Discounting , 2010 .

[103]  Paul F. Hill,et al.  Remember the Future: Working Memory Training Decreases Delay Discounting Among Stimulant Addicts , 2011, Biological Psychiatry.

[104]  Joseph T. McGuire,et al.  A Neural Signature of Hierarchical Reinforcement Learning , 2011, Neuron.

[105]  Jan Peters,et al.  The neural mechanisms of inter-temporal decision-making: understanding variability , 2011, Trends in Cognitive Sciences.

[106]  Matthijs A. A. van der Meer,et al.  Ventral striatum: a critical look at models of learning and evaluation , 2011, Current Opinion in Neurobiology.

[107]  Dylan A. Simon,et al.  Neural Correlates of Forward Planning in a Spatial Decision Task in Humans , 2011, The Journal of Neuroscience.

[108]  L. Green,et al.  Modeling the effect of reward amount on probability discounting. , 2011, Journal of the experimental analysis of behavior.

[109]  P. Dayan,et al.  Model-based influences on humans’ choices and striatal prediction errors , 2011, Neuron.

[110]  R. Benoit,et al.  A Neural Mechanism Mediating the Impact of Episodic Prospection on Farsighted Decisions , 2011, The Journal of Neuroscience.

[111]  Y. Niv,et al.  Ventral Striatum and Orbitofrontal Cortex Are Both Required for Model-Based, But Not Model-Free, Reinforcement Learning , 2011, The Journal of Neuroscience.

[112]  Samuel M. McClure,et al.  A mechanism for reducing delay discounting by altering temporal attention. , 2011, Journal of the experimental analysis of behavior.

[113]  Peter Dayan,et al.  Bonsai Trees in Your Head: How the Pavlovian System Sculpts Goal-Directed Choices by Pruning Decision Trees , 2012, PLoS Comput. Biol..

[114]  Karl J. Friston,et al.  Computational psychiatry , 2012, Trends in Cognitive Sciences.

[115]  Alan J Budney,et al.  Delay discounting predicts adolescent substance abuse treatment outcome. , 2012, Experimental and clinical psychopharmacology.

[116]  Karl J. Friston,et al.  Computational psychiatry (vol 16, pg 72, 2012) , 2012 .

[117]  Cognitive Control, Cognitive Search, and Motivational Salience: A Systems Neuroscience Approach , 2012 .

[118]  L. Green,et al.  Future decision‐making without episodic mental time travel , 2012, Hippocampus.

[119]  Matthijs A. A. van der Meer,et al.  Information Processing in Decision-Making Systems , 2012, The Neuroscientist : a review journal bringing neurobiology, neurology and psychiatry.