Interacting with volatile environments stabilizes hidden-state inference and its brain signatures

Making accurate decisions in uncertain environments requires identifying the generative cause of sensory cues, but also the expected outcomes of possible actions. Although both cognitive processes can be formalized as Bayesian inference, they are commonly studied using different experimental frameworks, making their formal comparison difficult. Here, by framing a reversal learning task either as cue-based or outcome-based inference, we found that humans perceive the same volatile environment as more stable when inferring its hidden state by interaction with uncertain outcomes than by observation of equally uncertain cues. Multivariate patterns of magnetoencephalo-graphic (MEG) activity reflected this behavioral difference in the neural interaction between inferred beliefs and incoming evidence, an effect originating from associative regions in the temporal lobe. Together, these findings indicate that the degree of control over the sampling of volatile environments shapes human learning and decision-making under uncertainty.

[1]  Jeff Miller,et al.  Measurement of ERP latency differences: a comparison of single-participant and jackknife-based scoring methods. , 2008, Psychophysiology.

[2]  M. Shadlen,et al.  Decision Making as a Window on Cognition , 2013, Neuron.

[3]  P. Dayan,et al.  A Bayesian formulation of behavioral control , 2009, Cognition.

[4]  Jeffrey M. Zacks,et al.  Searchlight analysis: Promise, pitfalls, and potential , 2013, NeuroImage.

[5]  Joseph W Kable,et al.  Normative evidence accumulation in unpredictable environments , 2015, eLife.

[6]  Colin Camerer,et al.  A framework for studying the neurobiology of value-based decision making , 2008, Nature Reviews Neuroscience.

[7]  R. Jardri,et al.  Circular inferences in schizophrenia. , 2013, Brain : a journal of neurology.

[8]  Peter R. Harris,et al.  Sufficient grounds for optimism? The relationship between perceived controllability and optimistic bias , 1996 .

[9]  A. Dale,et al.  Cortical Surface-Based Analysis II: Inflation, Flattening, and a Surface-Based Coordinate System , 1999, NeuroImage.

[10]  Andrew Y. Ng,et al.  Pharmacokinetics of a novel formulation of ivermectin after administration to goats , 2000, ICML.

[11]  Pieter Abbeel,et al.  Apprenticeship learning via inverse reinforcement learning , 2004, ICML.

[12]  Todd M Gureckis,et al.  Self-Directed Learning , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[13]  D. Markant,et al.  Is it better to select or to receive? Learning via active and passive hypothesis testing. , 2014, Journal of experimental psychology. General.

[14]  P. Haggard,et al.  Sense of agency , 2012, Current Biology.

[15]  Valentin Wyart,et al.  Choice variability and suboptimality in uncertain environments , 2016, Current Opinion in Behavioral Sciences.

[16]  Raymond J. Dolan,et al.  The anatomy of choice: active inference and agency , 2013, Front. Hum. Neurosci..

[17]  Timothy D. Hanks,et al.  Perceptual Decision Making in Rodents, Monkeys, and Humans , 2017, Neuron.

[18]  Bingni W. Brunton,et al.  Distinct effects of prefrontal and parietal cortex inactivations on an accumulation of evidence task in the rat , 2015, bioRxiv.

[19]  P. Glimcher,et al.  Midbrain Dopamine Neurons Encode a Quantitative Reward Prediction Error Signal , 2005, Neuron.

[20]  Timothy E. J. Behrens,et al.  Review Frontal Cortex and Reward-guided Learning and Decision-making Figure 1. Frontal Brain Regions in the Macaque Involved in Reward-guided Learning and Decision-making Finer Grained Anatomical Divisions with Frontal Cortical Systems for Reward-guided Behavior , 2022 .

[21]  Leslie G. Ungerleider,et al.  The neural systems that mediate human perceptual decision making , 2008, Nature Reviews Neuroscience.

[22]  Marius Usher,et al.  Decisions reduce sensitivity to subsequent information , 2015, Proceedings of the Royal Society B: Biological Sciences.

[23]  Richard M. Leahy,et al.  Electromagnetic brain mapping - IEEE Signal Processing Magazine , 2001 .

[24]  Marie Helweg-Larsen,et al.  Perceived Control and the Optimistic Bias: A Meta-Analytic Review , 2002 .

[25]  Justin L. Gardner,et al.  A Switching Observer for Human Perceptual Estimation , 2017, Neuron.

[26]  Timothy E. J. Behrens,et al.  Learning the value of information in an uncertain world , 2007, Nature Neuroscience.

[27]  Mel W. Khaw,et al.  Reminders of past choices bias decisions for reward in humans , 2017, Nature Communications.

[28]  Timothy Edward John Behrens,et al.  How Green Is the Grass on the Other Side? Frontopolar Cortex and the Evidence in Favor of Alternative Courses of Action , 2009, Neuron.

[29]  M. Lebreton,et al.  Behavioural and neural characterization of optimistic reinforcement learning , 2017, Nature Human Behaviour.

[30]  Karl J. Friston,et al.  Bayesian model selection for group studies , 2009, NeuroImage.

[31]  A. Doucet,et al.  Particle Markov chain Monte Carlo methods , 2010 .

[32]  R. Oostenveld,et al.  Nonparametric statistical testing of EEG- and MEG-data , 2007, Journal of Neuroscience Methods.

[33]  N. Daw,et al.  Hippocampal Contributions to Model-Based Planning and Spatial Memory , 2019, Neuron.

[34]  S. Gershman,et al.  Dopamine reward prediction errors reflect hidden state inference across time , 2017, Nature Neuroscience.

[35]  Robert Oostenveld,et al.  FieldTrip: Open Source Software for Advanced Analysis of MEG, EEG, and Invasive Electrophysiological Data , 2010, Comput. Intell. Neurosci..

[36]  Jan Drugowitsch,et al.  Computational Precision of Mental Inference as Critical Source of Human Choice Suboptimality , 2016, Neuron.

[37]  Timothy E. J. Behrens,et al.  Organizing conceptual knowledge in humans with a gridlike code , 2016, Science.

[38]  Richard M. Leahy,et al.  Brainstorm: A User-Friendly Application for MEG/EEG Analysis , 2011, Comput. Intell. Neurosci..

[39]  Philip L. Smith,et al.  A comparison of sequential sampling models for two-choice reaction time. , 2004, Psychological review.

[40]  Sylvain Baillet,et al.  Magnetoencephalography for brain electrophysiology and imaging , 2017, Nature Neuroscience.

[41]  N. Chater,et al.  Précis of Bayesian Rationality: The Probabilistic Approach to Human Reasoning , 2009, Behavioral and Brain Sciences.

[42]  F. D. de Lange,et al.  Action sharpens sensory representations of expected outcomes , 2018, Nature Communications.

[43]  Bingni W. Brunton,et al.  Distinct relationships of parietal and prefrontal cortices to evidence accumulation , 2014, Nature.

[44]  M. Shadlen,et al.  Decision Making and Sequential Sampling from Memory , 2016, Neuron.

[45]  Anders M. Dale,et al.  Cortical Surface-Based Analysis I. Segmentation and Surface Reconstruction , 1999, NeuroImage.

[46]  Gilles Faÿ,et al.  Características inmunológicas claves en la fisiopatología de la sepsis. Infectio , 2009 .

[47]  Jonathan D. Cohen,et al.  The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced-choice tasks. , 2006, Psychological review.

[48]  P. Rudebeck,et al.  The neural basis of reversal learning: An updated perspective , 2017, Neuroscience.

[49]  I. J. Myung,et al.  When a good fit can be bad , 2002, Trends in Cognitive Sciences.

[50]  Gareth O. Roberts,et al.  Examples of Adaptive MCMC , 2009 .

[51]  Joachim Gross,et al.  Good practice for conducting and reporting MEG research , 2013, NeuroImage.

[52]  Christoph W Korn,et al.  How unrealistic optimism is maintained in the face of reality , 2011, Nature Neuroscience.

[53]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 2005, IEEE Transactions on Neural Networks.

[54]  Alan Edelman,et al.  Julia: A Fresh Approach to Numerical Computing , 2014, SIAM Rev..

[55]  A. Pouget,et al.  Probabilistic brains: knowns and unknowns , 2013, Nature Neuroscience.

[56]  Jonathan W. Pillow,et al.  Dissociated functional significance of decision-related activity in the primate dorsal stream , 2016, Nature.

[57]  Renaud Jardri,et al.  Experimental evidence for circular inference in schizophrenia , 2017, Nature Communications.

[58]  D. Wolpert,et al.  Seeing what you want to see: priors for one's own actions represent exaggerated expectations of success , 2014, Front. Behav. Neurosci..

[59]  Richard M. Leahy,et al.  Electromagnetic brain mapping , 2001, IEEE Signal Process. Mag..

[60]  Peter Dayan,et al.  A Neural Substrate of Prediction and Reward , 1997, Science.

[61]  M. Botvinick,et al.  The hippocampus as a predictive map , 2016 .

[62]  J. Gold,et al.  The neural basis of decision making. , 2007, Annual review of neuroscience.

[63]  Susan G. Wardle,et al.  Decoding Dynamic Brain Patterns from Evoked Responses: A Tutorial on Multivariate Pattern Analysis Applied to Time Series Neuroimaging Data , 2016, Journal of Cognitive Neuroscience.

[64]  Raymond J Dolan,et al.  A map of abstract relational knowledge in the human hippocampal–entorhinal cortex , 2017, eLife.

[65]  Anne E. Urai,et al.  Confirmation Bias through Selective Overweighting of Choice-Consistent Evidence , 2018, Current Biology.

[66]  V. Wyart,et al.  Computational noise in reward-guided learning drives behavioral variability in volatile environments , 2019, Nature Neuroscience.