Model-based approaches to neuroimaging: combining reinforcement learning theory with fMRI data.

The combination of functional magnetic resonance imaging (fMRI) with computational models for a given cognitive process provides a powerful framework for testing hypotheses about the neural computations underlying such processes in the brain. Here, we outline the steps involved in implementing this approach with reference to the application of reinforcement learning (RL) models that can account for human choice behavior during value-based decision making. The model generates internal variables which can be used to construct fMRI predictor variables and regressed against individual subjects' fMRI data. The resulting regression coefficients reflect the strength of the correlation with blood oxygenation level dependent (BOLD) activity and the relevant internal variables from the model. In the second part of this review, we describe human neuroimaging studies that have employed this analysis strategy to identify brain regions involved in the computations mediating reward-related decision making. Copyright © 2010 John Wiley & Sons, Ltd. For further resources related to this article, please visit the WIREs website.

[1]  Peter Dayan,et al.  A Neural Substrate of Prediction and Reward , 1997, Science.

[2]  J. Gläscher,et al.  Determining a role for ventromedial prefrontal cortex in encoding action-based value signals during reward-related decision making. , 2009, Cerebral cortex.

[3]  N. Daw,et al.  Striatal Activity Underlies Novelty-Based Choice in Humans , 2008, Neuron.

[4]  P. Dayan,et al.  Cortical substrates for exploratory decisions in humans , 2006, Nature.

[5]  J. O'Doherty,et al.  Regret and its avoidance: a neuroimaging study of choice behavior , 2005, Nature Neuroscience.

[6]  R. Weale.  Vision. A Computational Investigation Into the Human Representation and Processing of Visual Information. David Marr , 1983 .

[7]  R. Rescorla.  A theory of pavlovian conditioning: The effectiveness of reinforcement and non-reinforcement , 1972 .

[8]  Kevin McCabe,et al.  Neural signature of fictive learning signals in a sequential investment task , 2007, Proceedings of the National Academy of Sciences.

[9]  Karl J. Friston,et al.  Free-energy and the brain , 2007, Synthese.

[10]  J. O'Doherty,et al.  Model‐Based fMRI and Its Application to Reward Learning and Decision Making , 2007, Annals of the New York Academy of Sciences.

[11]  J. Gläscher,et al.  Formal Learning Theory Dissociates Brain Regions with Different Temporal Integration , 2005, Neuron.

[12]  C. Frith,et al.  Meeting of minds: the medial frontal cortex and social cognition , 2006, Nature Reviews Neuroscience.

[13]  H. Simon,et al.  Models of Thought , 1979 .

[14]  Karl J. Friston,et al.  Temporal Difference Models and Reward-Related Learning in the Human Brain , 2003, Neuron.

[15]  P. Montague,et al.  Activity in human ventral striatum locked to errors of reward prediction , 2002, Nature Neuroscience.

[16]  Karl J. Friston,et al.  Dynamic causal modelling , 2003, NeuroImage.

[17]  B. Balleine,et al.  Goal-directed instrumental action: contingency and incentive learning and their cortical substrates , 1998, Neuropharmacology.

[18]  Peter Bossaerts,et al.  Neural correlates of mentalizing-related computations during strategic interactions in humans , 2008, Proceedings of the National Academy of Sciences.

[19]  Karl J. Friston,et al.  Nonlinear Dynamic Causal Models for Fmri Nonlinear Dynamic Causal Models for Fmri Nonlinear Dynamic Causal Models for Fmri , 2022 .

[20]  Mark W Woolrich,et al.  Associative learning of social value , 2008, Nature.

[21]  J. O'Doherty,et al.  Is Avoiding an Aversive Outcome Rewarding? Neural Substrates of Avoidance Learning in the Human Brain , 2006, PLoS biology.

[22]  R. Sugden,et al.  Regret Theory: An alternative theory of rational choice under uncertainty Review of Economic Studies , 1982 .

[23]  P. Dayan,et al.  Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control , 2005, Nature Neuroscience.

[24]  Timothy E. J. Behrens,et al.  Learning the value of information in an uncertain world , 2007, Nature Neuroscience.

[25]  Karl J. Friston,et al.  Dynamic causal modeling , 2010, Scholarpedia.

[26]  Karl J. Friston,et al.  A Dual Role for Prediction Error in Associative Learning , 2008, Cerebral cortex.

[27]  E. Vaadia,et al.  Midbrain dopamine neurons encode decisions for future action , 2006, Nature Neuroscience.

[28]  J. O'Doherty,et al.  The Role of the Ventromedial Prefrontal Cortex in Abstract State-Based Inference during Decision Making in Humans , 2006, The Journal of Neuroscience.

[29]  David E. Bell,et al.  Regret in Decision Making under Uncertainty , 1982, Oper. Res..

[30]  Richard S. Sutton,et al.  Reinforcement Learning , 1992, Handbook of Machine Learning.

[31]  P. Dayan,et al.  Choice values , 2006, Nature Neuroscience.

[32]  Samuel M. McClure,et al.  Temporal Prediction Errors in a Passive Learning Task Activate Human Striatum , 2003, Neuron.

[33]  J. O'Doherty,et al.  Dissociating Valence of Outcome from Behavioral Control in Human Orbital and Ventral Prefrontal Cortices , 2003, The Journal of Neuroscience.

[34]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.