Navigating through abstract decision spaces: Evaluating the role of state generalization in a dynamic decision-making task

Research on dynamic decision-making tasks, in which the payoffs associated with each choice vary with participants’ recent choice history, shows that humans have difficulty making long-term optimal choices in the presence of attractive immediate rewards. However, a number of recent studies have shown that simple cues providing information about the underlying state of the task environment may facilitate optimal responding. In this study, we examined the mechanism by which this state knowledge influences choice behavior. We examined the possibility that participants use state information in conjunction with changing payoffs to extrapolate payoffs in future states. We found support for this hypothesis in an experiment in which generalizations based on this state information worked to the benefit or detriment of task performance, depending on the task’s payoff structure.

[1]  L. Nadel,et al.  The Hippocampus as a Cognitive Map , 1978 .

[2]  R. Herrnstein,et al.  Utility maximization and melioration: Internalities in individual choice , 1993 .

[3]  Wayne D. Gray,et al.  Melioration Dominates Maximization: Stable Suboptimal Performance Despite Global Feedback , 2006 .

[4]  Bradley C. Love,et al.  Short-term gains, long-term pains: How cues about state aid learning in dynamic environments , 2009, Cognition.

[5]  A. Siegel,et al.  The development of spatial representations of large-scale environments. , 1975, Advances in child development and behavior.

[6]  R. Rosenfeld.  Nature , 2009, Otolaryngology--head and neck surgery : official journal of American Academy of Otolaryngology-Head and Neck Surgery.

[7]  B. A. Cartwright,et al.  How honey bees use landmarks to guide their return to a food source , 1982, Nature.

[8]  Howard Rachlin,et al.  Self-control: Beyond commitment. , 1995 .

[9]  Dana H. Ballard,et al.  Learning to perceive and act by trial and error , 1991, Machine Learning.

[10]  D. Shanks,et al.  A re‐examination of melioration and rational choice , 2002 .

[11]  L. Green,et al.  Discounting of delayed rewards: Models of individual choice. , 1995, Journal of the experimental analysis of behavior.

[12]  Samuel M. McClure,et al.  Short-term memory traces for action bias in human reinforcement learning , 2007, Brain Research.

[13]  Bob Remington,et al.  When more means less: factors affecting human self-control in a local versus global choice paradigm , 1999 .

[14]  Andrew G. Barto,et al.  Reinforcement learning , 1998 .

[15]  Gordon E Legge,et al.  Lost in virtual space: studies in human and ideal spatial navigation. , 2006, Journal of experimental psychology. Human perception and performance.

[16]  Richard S. Sutton,et al.  Dimensions of Reinforcement Learning , 1998 .

[17]  P. Dayan,et al.  Cortical substrates for exploratory decisions in humans , 2006, Nature.

[18]  I. Erev,et al.  On adaptation, maximization, and reinforcement learning among cognitive strategies. , 2005, Psychological review.

[19]  R. Passingham.  The hippocampus as a cognitive map J. O'Keefe & L. Nadel, Oxford University Press, Oxford (1978). 570 pp., £25.00 , 1979, Neuroscience.