Evidence for Model-based Computations in the Human Amygdala during Pavlovian Conditioning

Contemporary computational accounts of instrumental conditioning have emphasized a role for a model-based system in which values are computed with reference to a rich model of the structure of the world, and a model-free system in which values are updated without encoding such structure. Much less studied is the possibility of a similar distinction operating at the level of Pavlovian conditioning. In the present study, we scanned human participants while they participated in a Pavlovian conditioning task with a simple structure while measuring activity in the human amygdala using a high-resolution fMRI protocol. After fitting a model-based algorithm and a variety of model-free algorithms to the fMRI data, we found evidence for the superiority of a model-based algorithm in accounting for activity in the amygdala compared to the model-free counterparts. These findings support an important role for model-based algorithms in describing the processes underpinning Pavlovian conditioning, as well as providing evidence of a role for the human amygdala in model-based inference.

[1]  R. Rescorla,et al.  A theory of Pavlovian conditioning : Variations in the effectiveness of reinforcement and nonreinforcement , 1972 .

[2]  Nicholas Ayache,et al.  Non-parametric Diffeomorphic Image Registration with the Demons Algorithm , 2007, MICCAI.

[3]  J. O'Doherty,et al.  Neural coding of reward-prediction error signals during classical conditioning with attractive faces. , 2007, Journal of neurophysiology.

[4]  D. Blei,et al.  Context, learning, and extinction. , 2010, Psychological review.

[5]  P. Dayan,et al.  States versus Rewards: Dissociable Neural Prediction Error Signals Underlying Model-Based and Model-Free Reinforcement Learning , 2010, Neuron.

[6]  E. Murray,et al.  The amygdala and reward , 2002, Nature Reviews Neuroscience.

[7]  Joseph E LeDoux,et al.  Molecular Mechanisms of Fear Learning and Memory , 2011, Cell.

[8]  P. Dayan,et al.  Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control , 2005, Nature Neuroscience.

[9]  Aaron C. Courville,et al.  Bayesian theories of conditioning in a changing world , 2006, Trends in Cognitive Sciences.

[10]  P. Dayan,et al.  Decision theory, reinforcement learning, and the brain , 2008, Cognitive, affective & behavioral neuroscience.

[11]  Rebecca Elliott,et al.  Instrumental responding for rewards is associated with enhanced neuronal response in subcortical reward systems , 2004, NeuroImage.

[12]  H. Soininen,et al.  MR volumetric analysis of the human entorhinal, perirhinal, and temporopolar cortices. , 1998, AJNR. American journal of neuroradiology.

[13]  C. Bradshaw,et al.  The fear-inhibited light reflex: importance of the anticipation of an aversive event. , 2004, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[14]  Hugo D Critchley,et al.  Emotional and autonomic consequences of spinal cord injury explored using functional brain imaging. , 2006, Brain : a journal of neurology.

[15]  Denis Dooley,et al.  Atlas of the Human Brain. , 1971 .

[16]  Jean-Philippe Thirion,et al.  Image matching as a diffusion process: an analogy with Maxwell's demons , 1998, Medical Image Anal..

[17]  J. O'Doherty,et al.  Appetitive and Aversive Olfactory Learning in Humans Studied Using Event-Related Functional Magnetic Resonance Imaging , 2002, The Journal of Neuroscience.

[18]  Joseph E LeDoux,et al.  Why We Think Plasticity Underlying Pavlovian Fear Conditioning Occurs in the Basolateral Amygdala , 1999, Neuron.

[19]  Karl J. Friston,et al.  Bayesian model selection for group studies , 2009, NeuroImage.

[20]  Karl J. Friston,et al.  Opponent appetitive-aversive neural processes underlie predictive learning of pain relief , 2005, Nature Neuroscience.

[21]  P. Dayan,et al.  Model-based influences on humans’ choices and striatal prediction errors , 2011, Neuron.

[22]  K. Doya,et al.  Understanding Neural Coding through the Model-Based Analysis of Decision Making , 2007, The Journal of Neuroscience.

[23]  J. O'Doherty,et al.  The Role of the Ventromedial Prefrontal Cortex in Abstract State-Based Inference during Decision Making in Humans , 2006, The Journal of Neuroscience.

[24]  E. Murray.  The amygdala, reward and emotion , 2007, Trends in Cognitive Sciences.

[25]  Can Ceritoglu,et al.  Increasing the power of functional maps of the medial temporal lobe by using large deformation diffeomorphic metric mapping. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[26]  J. Gläscher,et al.  Dissociable Systems for Gain- and Loss-Related Value Predictions and Errors of Prediction in the Human Brain , 2006, The Journal of Neuroscience.

[27]  W. K. Simmons,et al.  Circular analysis in systems neuroscience: the dangers of double dipping , 2009, Nature Neuroscience.

[28]  N. Daw,et al.  Differential roles of human striatum and amygdala in associative learning , 2011, Nature Neuroscience.

[29]  Guillem R. Esber,et al.  Neural Correlates of Variations in Event Processing during Learning in Basolateral Amygdala , 2010, The Journal of Neuroscience.

[30]  Michael A Yassa,et al.  A quantitative evaluation of cross-participant registration techniques for MRI studies of the medial temporal lobe , 2009, NeuroImage.

[31]  M. Delgado,et al.  Extending animal models of fear conditioning to humans , 2006, Biological Psychology.

[32]  G H Glover,et al.  Image‐based method for retrospective correction of physiological motion effects in fMRI: RETROICOR , 2000, Magnetic resonance in medicine.

[33]  R. Dolan,et al.  Classical fear conditioning in functional neuroimaging , 2000, Current Opinion in Neurobiology.

[34]  B. Balleine,et al.  The Neural Mechanisms Underlying the Influence of Pavlovian Cues on Human Decision Making , 2008, The Journal of Neuroscience.

[35]  J. O'Doherty,et al.  Encoding Predictive Reward Value in Human Amygdala and Orbitofrontal Cortex , 2003, Science.

[36]  J. O'Doherty,et al.  Contributions of the Amygdala to Reward Expectancy and Choice Signals in Human Prefrontal Cortex , 2007, Neuron.

[37]  P. Dayan,et al.  Differential Encoding of Losses and Gains in the Human Striatum , 2007, The Journal of Neuroscience.

[38]  J. Pearce,et al.  A model for Pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli. , 1980, Psychological review.

[39]  G. Paxinos,et al.  Atlas of the Human Brain , 2000 .

[40]  S. Kakade,et al.  Learning and selective attention , 2000, Nature Neuroscience.

[41]  Ryan K. Jessup,et al.  Differentiable contributions of human amygdalar subregions in the computations underlying reward and avoidance learning , 2011, The European journal of neuroscience.

[42]  J. O'Doherty,et al.  Model‐Based fMRI and Its Application to Reward Learning and Decision Making , 2007, Annals of the New York Academy of Sciences.

[43]  Karl J. Friston,et al.  Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning , 2004, Science.

[44]  M. Bradley.  Natural selective attention: orienting and emotion. , 2009, Psychophysiology.

[45]  B. C. Lacey,et al.  Pupillary and cardiac activity during visual attention. , 1973, Psychophysiology.

[46]  Mitsuo Kawato,et al.  Multiple Model-Based Reinforcement Learning , 2002, Neural Computation.

[47]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .