Rules for aversive learning and decision-making

Associative and reinforcement learning rules describe how animals, including humans, predict events in the world. In aversive learning, these rules describe how we predict danger. Here we outline major associative and reinforcement rules for aversive learning as well as behavioural approaches to study them. We identify key findings from the emerging literature on how these different learning rules are instantiated in the mammalian brain. We also highlight key areas where understanding is lacking and more research is needed. Finally, we consider how these learning rules not only help solve the problem of predicting danger but also provide a theoretical and empirical basis for understanding aversive decision-making.

[1]  R. Paz,et al.  Functional Connectivity between Amygdala and Cingulate Cortex for Adaptive Aversive Learning , 2013, Neuron.

[2]  R. Rescorla Reduction in the effectiveness of reinforcement after prior excitatory conditioning , 1970 .

[3]  Gavan P McNally,et al.  Blocking, unblocking, and overexpectation of fear: a role for opioid receptors in the regulation of Pavlovian association formation. , 2004, Behavioral neuroscience.

[4]  G. McNally,et al.  Opioid Receptors in the Nucleus Accumbens Regulate Attentional Learning in the Blocking Paradigm , 2006, The Journal of Neuroscience.

[5]  Jonathan Yi,et al.  Endogenous opioids regulate social threat learning in humans , 2017, Nature Communications.

[6]  M. Pelley The Role of Associative History in Models of Associative Learning: A Selective Review and a Hybrid Model: , 2004 .

[7]  G. McNally,et al.  Opioid receptors in the midbrain periaqueductal gray regulate prediction errors during pavlovian fear conditioning. , 2006, Behavioral neuroscience.

[8]  P. Dayan,et al.  Reinforcement learning: The Good, The Bad and The Ugly , 2008, Current Opinion in Neurobiology.

[9]  Christian Büchel,et al.  Blockade of Endogenous Opioid Neurotransmission Enhances Acquisition of Conditioned Fear in Humans , 2008, The Journal of Neuroscience.

[10]  R. Richardson,et al.  US habituation, like CS extinction, produces a decrement in conditioned fear responding that is NMDA dependent and subject to renewal and reinstatement , 2010, Neurobiology of Learning and Memory.

[11]  E. E. Bagley,et al.  Effects of chemogenetic excitation or inhibition of the ventrolateral periaqueductal gray on the acquisition and extinction of Pavlovian fear conditioning , 2017, Neurobiology of Learning and Memory.

[12]  P. Dayan,et al.  Model-based and model-free Pavlovian reward learning: Revaluation, revision, and revelation , 2014, Cognitive, affective & behavioral neuroscience.

[13]  Unblocking in Pavlovian fear conditioning. , 2008, Journal of experimental psychology. Animal behavior processes.

[14]  G. McNally,et al.  A role of nucleus accumbens dopamine receptors in the nucleus accumbens core, but not shell, in fear prediction error. , 2015, Behavioral neuroscience.

[15]  Y. Niv,et al.  Ventral Striatum and Orbitofrontal Cortex Are Both Required for Model-Based, But Not Model-Free, Reinforcement Learning , 2011, The Journal of Neuroscience.

[16]  J. Pearce,et al.  Effect of changing the unconditioned stimulus on appetitive blocking. , 1988, Journal of experimental psychology. Animal behavior processes.

[17]  B. Balleine,et al.  Motivational state controls the prediction error in Pavlovian appetitive-aversive interactions , 2018, Neurobiology of Learning and Memory.

[18]  R. Rescorla Learning about qualitatively different outcomes during a blocking procedure , 1999 .

[19]  J. Pearce,et al.  A model for Pavlovian learning: Variations in the effectiveness of conditioned but not of unconditioned stimuli. , 1980 .

[20]  Gavan P McNally,et al.  Disrupted Prediction Error Links Excessive Amygdala Activation to Excessive Fear , 2016, The Journal of Neuroscience.

[21]  G. McNally,et al.  Selecting danger signals: dissociable roles of nucleus accumbens shell and core glutamate in predictive fear learning , 2015, The European journal of neuroscience.

[22]  I. Weiner The "two-headed" latent inhibition model of schizophrenia: modeling positive and negative symptoms and their treatment , 2003, Psychopharmacology.

[23]  Gavan P McNally,et al.  Basolateral Amygdala Neurons Maintain Aversive Emotional Salience , 2017, The Journal of Neuroscience.

[24]  D. Silove,et al.  A Study of the Protective Function of Acute Morphine Administration on Subsequent Posttraumatic Stress Disorder , 2009, Biological Psychiatry.

[25]  Richard S. Sutton,et al.  Learning to predict by the methods of temporal differences , 1988, Machine Learning.

[26]  R. Rescorla Effect of US habituation following conditioning. , 1973, Journal of comparative and physiological psychology.

[27]  G. McNally,et al.  Opioid receptors mediate direct predictive fear learning: evidence from one-trial blocking. , 2007, Learning & memory.

[28]  A. R. Wagner,et al.  Dissociation of the blocking of conditioned eyeblink and conditioned fear following a shift in US locus , 1996 .

[29]  Joanna Oi-Yue Yau,et al.  Pharmacogenetic Excitation of Dorsomedial Prefrontal Cortex Restores Fear Prediction Error , 2015, The Journal of Neuroscience.

[30]  S. Killcross,et al.  The prelimbic cortex contributes to the down-regulation of attention toward redundant cues. , 2014, Cerebral cortex.

[31]  A. Arntz,et al.  Opioid antagonist affects behavioral effects of exposure in vivo. , 1993, Journal of consulting and clinical psychology.

[32]  Mihaela D Iordanova,et al.  Dopamine activity in the nucleus accumbens modulates blocking in fear conditioning , 2006, The European journal of neuroscience.

[33]  G. McNally,et al.  Predicting danger: the nature, consequences, and neural mechanisms of predictive fear learning. , 2006, Learning & memory.

[34]  Hiu Tin Leung,et al.  Two ways to deepen extinction and the difference between them. , 2012, Journal of experimental psychology. Animal behavior processes.

[35]  N. Mackintosh A Theory of Attention: Variations in the Associability of Stimuli with Reinforcement , 1975 .

[36]  Stephen Maren,et al.  Associative structure of fear memory after basolateral amygdala lesions in rats. , 2008, Behavioral neuroscience.

[37]  C. Taylor,et al.  Opioid antagonist impedes exposure. , 1991, Journal of consulting and clinical psychology.

[38]  Peter Dayan,et al.  Temporal difference models describe higher-order learning in humans , 2004, Nature.

[39]  J. Zuber,et al.  Dorsal tegmental dopamine neurons gate associative learning of fear , 2018, Nature Neuroscience.

[40]  R. Rescorla Deepened extinction from compound stimulus presentation. , 2006, Journal of experimental psychology. Animal behavior processes.

[41]  L. Kamin Attention-like processes in classical conditioning , 1967 .

[42]  J. Konorski Integrative activity of the brain : an interdisciplinary approach , 1967 .

[43]  G. McNally,et al.  Appetitive-aversive interactions in Pavlovian fear conditioning. , 2012, Behavioral neuroscience.

[44]  R. Rescorla,et al.  The effect of change in US quality on the blocking effect , 1974, The Pavlovian journal of biological science.

[45]  W. Schultz Getting Formal with Dopamine and Reward , 2002, Neuron.

[46]  G. McNally,et al.  Temporal-difference prediction errors and Pavlovian fear conditioning: role of NMDA and opioid receptors. , 2007, Behavioral neuroscience.

[47]  H. T. Blair,et al.  Placing prediction into the fear circuit , 2011, Trends in Neurosciences.

[48]  Gavan P. McNally,et al.  A role for midline and intralaminar thalamus in the associative blocking of Pavlovian fear conditioning , 2014, Front. Behav. Neurosci..

[49]  H. T. Blair,et al.  Neural substrates for expectation-modulated fear learning in the amygdala and periaqueductal gray , 2010, Nature Neuroscience.

[50]  A. R. Wagner,et al.  Evolution of a structured connectionist model of Pavlovian conditioning (AESOP). , 1989 .

[51]  Edgar A Ycu,et al.  A feedback neural circuit for calibrating aversive memory strength , 2016, Nature Neuroscience.

[52]  Cyril Herry,et al.  Encoding of fear learning and memory in distributed neuronal circuits , 2014, Nature Neuroscience.

[53]  K. Schneider,et al.  Naltrexone renders one-session exposure therapy less effective: a controlled pilot study. , 2007, Journal of anxiety disorders.

[54]  R. Rescorla Effect of inflation of the unconditioned stimulus value following conditioning. , 1974 .

[55]  S. Killcross,et al.  Behavioral and neurobiological mechanisms of punishment: implications for psychiatric disorders , 2018, Neuropsychopharmacology.

[56]  G. McNally,et al.  Complementary roles for amygdala and periaqueductal gray in temporal-difference fear learning. , 2008, Learning & memory.

[57]  E. E. Bagley,et al.  Brief optogenetic inhibition of rat lateral or ventrolateral periaqueductal gray augments the acquisition of Pavlovian fear conditioning , 2017, bioRxiv.

[58]  Daphna Shohamy,et al.  Representation of aversive prediction errors in the human periaqueductal gray , 2014, Nature Neuroscience.

[59]  D. Glanzman,et al.  Commonalities and Differences in the Substrates Underlying Consolidation of First- and Second-Order Conditioned Fear , 2018, The Journal of Neuroscience.

[60]  Guillem R. Esber,et al.  Surprise! Neural correlates of Pearce–Hall and Rescorla–Wagner coexist within the brain , 2012, The European journal of neuroscience.

[61]  R. Rescorla,et al.  A theory of Pavlovian conditioning : Variations in the effectiveness of reinforcement and nonreinforcement , 1972 .

[62]  P. Holland,et al.  Variations in unconditioned stimulus processing in unblocking. , 2005, Journal of experimental psychology. Animal behavior processes.

[63]  R. Dolan,et al.  The neurobiology of punishment , 2007, Nature Reviews Neuroscience.

[64]  R. Rescorla Are associative changes in acquisition and extinction negatively accelerated? , 2001, Journal of experimental psychology. Animal behavior processes.