Stress-sensitive inference of task controllability

Estimating environmental controllability enables agents to better predict upcoming events and decide when to engage controlled action selection. How does the human brain estimate environmental controllability? Trial-by-trial analysis of choices, decision times and neural activity in an explore-and-predict task demonstrate that humans solve this problem by comparing the predictions of an “actor” model with those of a reduced “spectator” model of their environment. Neural BOLD responses within striatal and medial prefrontal areas tracked the instantaneous difference in the prediction errors generated by these two statistical learning models. BOLD activity in the posterior cingulate, parietal and prefrontal cortices covaried with changes in estimated controllability. Exposure to inescapable stressors biased controllability estimates downward and increased reliance on the spectator model in an anxiety-dependent fashion. Taken together, these findings provide a mechanistic account of controllability inference and its distortion by stress exposure.

[1]  Y. Niv.  Learning task-state representations , 2019, Nature Neuroscience.

[2]  Karl J. Friston,et al.  Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning , 2004, Science.

[3]  Jordi Grau-Moya,et al.  A Unified Bellman Optimality Principle Combining Reward Maximization and Empowerment , 2019, NeurIPS.

[4]  David E. Warren,et al.  Damage to the Ventromedial Prefrontal Cortex Impairs Learning from Observed Outcomes , 2015, Cerebral cortex.

[5]  Michael Breakspear,et al.  Reconfiguration of functional brain networks and metabolic cost converge during task performance , 2020, eLife.

[6]  Jane R. Garrison,et al.  Prediction error in reinforcement learning: A meta-analysis of neuroimaging studies , 2013, Neuroscience & Biobehavioral Reviews.

[7]  Asaf Gilboa,et al.  Neurobiology of Schemas and Schema-Mediated Memory , 2017, Trends in Cognitive Sciences.

[8]  J. O'Doherty,et al.  Neural Correlates of the Divergence of Instrumental Probability Distributions , 2013, The Journal of Neuroscience.

[9]  S. Manohar,et al.  Dopamine promotes instrumental motivation, but reduces reward-related vigour , 2020, bioRxiv.

[10]  Francisco Ferreira,et al.  Theory and Evidence , 2009 .

[11]  M. Paluš,et al.  Inferring the directionality of coupling with conditional mutual information. , 2008, Physical review. E, Statistical, nonlinear, and soft matter physics.

[12]  Shannon M. Pruden,et al.  A Meta-analysis of Neuroimaging Studies , 2015 .

[13]  Lionel Rigoux,et al.  VBA: A Probabilistic Treatment of Nonlinear Models for Neurobiological and Behavioural Data , 2014, PLoS Comput. Biol..

[14]  L. Schwabe,et al.  Subjective Uncontrollability over Aversive Events Reduces Working Memory Performance and Related Large-Scale Network Interactions. , 2019, Cerebral cortex.

[15]  Shinsuke Shimojo,et al.  Neural Computations Underlying Arbitration between Model-Based and Model-free Learning , 2013, Neuron.

[16]  C. Hartley,et al.  Agency and the Calibration of Motivated Behavior , 2017, Trends in Cognitive Sciences.

[17]  M. Seligman,et al.  Learned helplessness: Theory and evidence. , 1976 .

[18]  Michael J. Frank,et al.  Faculty Opinions recommendation of States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. , 2010 .

[19]  M. Brass,et al.  Feeling in control: Neural correlates of experience of agency , 2013, Cortex.

[20]  Martin N. Hebart,et al.  The Decoding Toolbox (TDT): a versatile software package for multivariate analyses of functional imaging data , 2015, Front. Neuroinform..

[21]  Berit Brummerloh,et al.  Impaired awareness of action-outcome contingency and causality during healthy ageing and following ventromedial prefrontal cortex lesions , 2018, Neuropsychologia.

[22]  Samuel J. Gershman,et al.  Controllability governs the balance between Pavlovian and instrumental action selection , 2019, Nature Communications.

[23]  M. Cooper,et al.  Overlapping neurobiology of learned helplessness and conditioned defeat: Implications for PTSD and mood disorders , 2012, Neuropharmacology.

[24]  Marcel Brass,et al.  Was it me or was it you? How the sense of agency originates from ideomotor learning revealed by fMRI , 2009, NeuroImage.

[25]  Karl J. Friston,et al.  Canonical Microcircuits for Predictive Coding , 2012, Neuron.

[26]  S. Maier,et al.  Medial prefrontal cortex determines how stressor controllability affects behavior and dorsal raphe nucleus , 2005, Nature Neuroscience.

[27]  Michael Moutoussis,et al.  Improving the reliability of model-based decision-making estimates in the two-stage decision task with reaction-times and drift-diffusion modeling , 2019, PLoS Comput. Biol..

[28]  P. Dayan,et al.  States versus Rewards: Dissociable Neural Prediction Error Signals Underlying Model-Based and Model-Free Reinforcement Learning , 2010, Neuron.

[29]  Sang Wan Lee,et al.  Task complexity interacts with state-space uncertainty in the arbitration between model-based and model-free learning , 2019, Nature Communications.

[30]  M. Desmurget,et al.  The dorsal medial frontal cortex mediates automatic motor inhibition in uncertain contexts: Evidence from combined fMRI and EEG studies , 2014, Human brain mapping.

[31]  Christopher I. Moore,et al.  Dopamine waves as a mechanism for spatiotemporal credit assignment , 2019, bioRxiv.

[32]  Matthew R Nassar,et al.  Individual Neurons in the Cingulate Cortex Encode Action Monitoring, Not Selection, during Adaptive Decision-Making , 2019, The Journal of Neuroscience.

[33]  S. Williams,et al.  Instability of default mode network connectivity in major depression: a two-sample confirmation study , 2017, Translational Psychiatry.

[34]  M. Paulus,et al.  Increased anterior insula activity in anxious individuals is linked to diminished perceived control , 2015, Translational Psychiatry.

[35]  H. Flor,et al.  Effects of stressor controllability on psychophysiological, cognitive and behavioural responses in patients with major depression and dysthymia , 2008, Psychological Medicine.

[36]  P. Haggard.  Sense of agency in the human brain , 2017, Nature Reviews Neuroscience.

[37]  M. Seligman,et al.  Learned helplessness at fifty: Insights from neuroscience. , 2016, Psychological review.

[38]  D. Sharp,et al.  The role of the posterior cingulate cortex in cognition and disease. , 2014, Brain : a journal of neurology.

[39]  Rajesh P. N. Rao,et al.  Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. , 1999 .

[40]  Jessica I. Määttä,et al.  Dopamine promotes cognitive effort by biasing the benefits versus costs of cognitive work , 2019, Science.

[41]  M. Delgado,et al.  A Reward-Based Framework of Perceived Control , 2019, Front. Neurosci..

[42]  Jan Drugowitsch,et al.  Interacting with volatile environments stabilizes hidden-state inference and its brain signatures , 2019, Nature Communications.

[43]  D. McLaren,et al.  Controllability Modulates the Anticipatory Response in the Human Ventromedial Prefrontal Cortex , 2012, Front. Psychology.

[44]  P. Dayan,et al.  A Bayesian formulation of behavioral control , 2009, Cognition.

[45]  M. Harrow,et al.  Locus of control: Relation to schizophrenia, to recovery, and to depression and psychosis — A 15-year longitudinal study , 2009, Psychiatry Research.

[46]  Miriam C Klein-Flügge,et al.  Multiple associative structures created by reinforcement and incidental statistical learning mechanisms , 2019, Nature Communications.

[47]  Susanne Becker,et al.  Different Brain Circuitries Mediating Controllable and Uncontrollable Pain , 2016, The Journal of Neuroscience.

[48]  Cecilia Cheng,et al.  Cultural meaning of perceived control: a meta-analysis of locus of control and psychological symptoms across 18 cultural regions. , 2013, Psychological bulletin.

[49]  L. Luo,et al.  Deep posteromedial cortical rhythm in dissociation , 2020, Nature.

[50]  A. Seth,et al.  Granger causality and transfer entropy are equivalent for Gaussian variables. , 2009, Physical review letters.

[51]  T. Robbins,et al.  Obsessive–compulsive disorder patients have a reduced sense of control on the illusion of control task , 2014, Front. Psychol..

[52]  Alice Y. Chiang,et al.  Working-memory capacity protects model-based learning from stress , 2013, Proceedings of the National Academy of Sciences.

[53]  S. Maier,et al.  Stressor Controllability Modulates Stress-Induced Dopamine and Serotonin Efflux and Morphine-Induced Serotonin Efflux in the Medial Prefrontal Cortex , 2003, Neuropsychopharmacology.

[54]  Simone Kühn,et al.  In and out of control: brain mechanisms linking fluency of action selection to self-agency in patients with schizophrenia , 2017, Brain : a journal of neurology.

[55]  Jonathan D. Cohen,et al.  Dorsal anterior cingulate cortex and the value of control , 2016, Nature Neuroscience.

[56]  Michael Moutoussis,et al.  Credit assignment to state-independent task representations and its relationship with model-based decision making , 2019, Proceedings of the National Academy of Sciences.

[57]  S. Dickerson,et al.  Acute stressors and cortisol responses: a theoretical integration and synthesis of laboratory research. , 2004, Psychological bulletin.

[58]  Shakir Mohamed,et al.  Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning , 2015, NIPS.

[59]  Fair M. Vassoler,et al.  Raphe GABAergic Neurons Mediate the Acquisition of Avoidance after Social Defeat , 2013, The Journal of Neuroscience.

[60]  Muriah D. Wheelock,et al.  Controllability modulates the neural response to predictable but not unpredictable threat in humans , 2015, NeuroImage.

[61]  Agnès Gruart,et al.  Observational learning in mice can be prevented by medial prefrontal cortex stimulation and enhanced by nucleus accumbens stimulation. , 2012, Learning & memory.