Beyond simple model-free reinforcement learning in human decision making

Over the last two decades, there has been a large scale effort in cognitive neuroscience to understand learning and decision making from the perspective of simple model-free reinforcement learning algorithms. This interest was invigorated in the mid 1990’s, when it was realized that the phasic activity of midbrain dopaminergic neurons resembles reward prediction errors. The algorithms studied formalize the notion of learning from past experiences through trial and error. Although important, there are many aspects of behavior they cannot explain. More recent work has begun to fill in some of these gaps by borrowing yet additional ideas from computational reinforcement learning. One line of inquiry has concentrated on aligning goal-directed behavior, which resembles the common sense notion of “planning”, with model-based reinforcement learning. This work has aimed to understand how the brain is able to learn the world model prescribed by the model-based framework, and to characterize the neural correlates of the value functions it predicts. This thesis adds to this work by offering two separate, but related, algorithmic accounts of how the brain may be able to actually map the world model into a decision. Existing data are examined and new experiments are performed. A second line of inquiry has concentrated on understanding behavior from the perspective of hierarchical reinforcement learning. The thesis makes two contributions to this area as well. First, it is shown that the brain codes pseudo-reward prediction errors, a prediction error in response to a faux reward signal that is used to train skills that are not in themselves useful, but that may be used to achieve other means. Second, an optimality framework is provided for understanding which skills are most beneficial to have when confronted with an ensemble of tasks.