Formalizing the Function of Anterior Insula in Rapid Adaptation

Anterior insula (aIns) is thought to play a crucial role in rapid adaptation in an ever-changing environment. Mathematically, it is known to track risk and surprise. Modern theories of learning, however, assign a dominant role to signed prediction errors (PEs), not to risk and surprise. Risk and surprise only enter to the extent that they modulate the learning rate, in an attempt to approximate Bayesian learning. Even without such modulation, adaptation is still possible, albeit slow. Here, I propose a new theory of learning, reference-model based learning (RMBL), where risk and surprise are central, and PEs play a secondary, though still crucial, role. The primary goal is to bring outcomes in line with expectations in the reference model (RM). Learning is modulated by how large the PEs are relative to model anticipation, i.e., to surprise as defined by the RM. In a target location prediction task where participants were continuously required to adapt, choices appeared to be closer with to RMBL predictions than to Bayesian learning. aIns reaction to surprise was more acute in the more difficult treatment, consistent with its hypothesized role in metacognition. I discuss links with related theories, such as Active Inference, Actor-Critic Models and Reference-Model Based Adaptive Control.

[1]  Britt Anderson,et al.  Rejecting Outliers: Surprising Changes Do Not Always Improve Belief Updating , 2018, Decision.

[2]  G. McCarthy,et al.  Decisions under Uncertainty: Probabilistic Context Influences Activation of Prefrontal and Parietal Cortices , 2005, The Journal of Neuroscience.

[3]  Timothy E. J. Behrens,et al.  Learning the value of information in an uncertain world , 2007, Nature Neuroscience.

[4]  Evan M. Gordon,et al.  Neural Signatures of Economic Preferences for Risk and Ambiguity , 2006, Neuron.

[5]  Nozer D. Singpurwalla,et al.  Understanding the Kalman Filter , 1983 .

[6]  D. Freedman,et al.  On the consistency of Bayes estimates , 1986 .

[7]  A. Craig,et al.  How do you feel — now? The anterior insula and human awareness , 2009, Nature Reviews Neuroscience.

[8]  Peter Bossaerts,et al.  The Neural Representation of Unexpected Uncertainty during Value-Based Decision Making , 2013, Neuron.

[9]  Peter Stone,et al.  Reinforcement learning , 2019, Scholarpedia.

[10]  A. Borst.  Seeing smells: imaging olfactory learning in bees , 1999, Nature Neuroscience.

[11]  Kenneth Hugdahl,et al.  Prediction of human errors by maladaptive changes in event-related brain networks , 2008, Proceedings of the National Academy of Sciences.

[12]  S. Michael Malinconico,et al.  Decisions under Uncertainty. , 1984 .

[13]  S. Quartz,et al.  Human Insula Activation Reflects Risk Prediction Errors As Well As Risk , 2008, The Journal of Neuroscience.

[14]  M. Philiastides,et al.  Separate neural representations of prediction error valence and surprise: Evidence from an fMRI meta‐analysis , 2018, Human brain mapping.

[15]  K. Preuschoff,et al.  Adding Prediction Risk to the Theory of Reward Learning , 2007, Annals of the New York Academy of Sciences.

[16]  B. Anderson,et al.  Right hemisphere brain damage impairs strategy updating. , 2012, Cerebral cortex.

[17]  Robert C. Wilson,et al.  An Approximately Bayesian Delta-Rule Model Explains the Dynamics of Belief Updating in a Changing Environment , 2010, The Journal of Neuroscience.

[18]  Karl J. Friston.  Active inference and agency , 2014, Cognitive neuroscience.

[19]  P. Dayan,et al.  States versus Rewards: Dissociable Neural Prediction Error Signals Underlying Model-Based and Model-Free Reinforcement Learning , 2010, Neuron.

[20]  A. Craig.  Significance of the insula for the evolution of human awareness of feelings from the body. , 2011, Annals of the New York Academy of Sciences.

[21]  R. Dolan,et al.  The neural basis of metacognitive ability , 2012, Philosophical Transactions of the Royal Society B: Biological Sciences.

[22]  Peter Dayan,et al.  Expected and Unexpected Uncertainty: ACh and NE in the Neocortex , 2002, NIPS.

[23]  Peter Dayan,et al.  A Neural Substrate of Prediction and Reward , 1997, Science.

[24]  Zhong-Lin Lu,et al.  Neural correlates of risk prediction error during reinforcement learning in humans , 2009, NeuroImage.

[25]  Monique Ernst,et al.  Decision-making in a Risk-taking Task: A PET Study , 2002, Neuropsychopharmacology.

[26]  Karl J. Friston,et al.  Active inference and agency: optimal control without cost functions , 2012, Biological Cybernetics.

[27]  H. Critchley,et al.  Neural Activity in the Human Brain Relating to Uncertainty and Arousal during Anticipation , 2001, Neuron.

[28]  Joseph T. McGuire,et al.  Functionally Dissociable Influences on Learning Rate in a Dynamic Environment , 2014, Neuron.

[29]  P. Dayan,et al.  Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control , 2005, Nature Neuroscience.

[30]  Karen J. Mullinger,et al.  Spatiotemporal neural characterization of prediction error valence and surprise during reward learning in humans , 2017, Scientific Reports.

[31]  Corianne Rogalsky,et al.  Increased activation in the right insula during risk-taking decision making is related to harm avoidance and neuroticism , 2003, NeuroImage.

[32]  Norbert Kathmann,et al.  Neural correlates of error awareness , 2007, NeuroImage.

[33]  H. Simon,et al.  Models Of Man : Social And Rational , 1957 .

[34]  J. Pearce,et al.  A model for Pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli. , 1980, Psychological review.

[35]  H. Simon,et al.  Theories of Decision-Making in Economics and Behavioural Science , 1966 .

[36]  Richard S. Sutton,et al.  Adapting Bias by Gradient Descent: An Incremental Version of Delta-Bar-Delta , 1992, AAAI.

[37]  J. O'Doherty,et al.  The Role of the Ventromedial Prefrontal Cortex in Abstract State-Based Inference during Decision Making in Humans , 2006, The Journal of Neuroscience.

[38]  P. Bossaerts,et al.  Neural Mechanisms Behind Identification of Leptokurtic Noise and Adaptive Behavioral Response , 2016, Cerebral cortex.

[39]  Karl J. Friston.  The free-energy principle: a unified brain theory? , 2010, Nature Reviews Neuroscience.

[40]  Karl J. Friston,et al.  Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning , 2004, Science.

[41]  Karl J. Friston,et al.  Reinforcement Learning or Active Inference? , 2009, PloS one.

[42]  Karl J. Friston,et al.  Temporal Difference Models and Reward-Related Learning in the Human Brain , 2003, Neuron.