Electrophysiological correlates of state transition prediction errors

Planning behavior depends crucially on the ability to distinguish between the likely and unlikely consequences of an action. Formal computational models of planning postulate the existence of a neural mechanism that tracks the transition model of the environment, i.e., a model that explicitly represents the probabilities of action consequences. However, empirical findings relating to such a mechanism are scarce. Here we report the results of two electroencephalographic experiments examining the neural correlates of transition model learning. The results implicate fronto-midline theta and delta oscillations in this process and suggest a role of the anterior midcingulate cortex in planning behavior.

[1]  Adam Johnson,et al.  Neural Ensembles in CA3 Transiently Encode Paths Forward of the Animal at a Decision Point , 2007, The Journal of Neuroscience.

[2]  Michael X. Cohen,et al.  Reward expectation modulates feedback-related negativity and EEG spectra , 2007, NeuroImage.

[3]  B. Balleine,et al.  The algorithmic neuroanatomy of action-outcome learning , 2017, bioRxiv.

[4]  Clay B. Holroyd,et al.  Sensitivity of electrophysiological activity from medial frontal cortex to utilitarian and performance feedback. , 2004, Cerebral cortex.

[5]  Clay B. Holroyd,et al.  Brain mechanisms underlying apathy , 2018, Journal of Neurology, Neurosurgery, and Psychiatry.

[6]  Clay B. Holroyd,et al.  Motivation of extended behaviors by anterior cingulate cortex , 2012, Trends in Cognitive Sciences.

[7]  A. Markman,et al.  The Curse of Planning: Dissecting Multiple Reinforcement-Learning Systems by Taxing the Central Executive , 2013 .

[8]  B. Balleine,et al.  The Role of Learning in the Operation of Motivational Systems , 2002 .

[9]  Peter Bossaerts,et al.  The Neural Representation of Unexpected Uncertainty during Value-Based Decision Making , 2013, Neuron.

[10]  Markus Ullsperger,et al.  Surprise and Error: Common Neuronal Architecture for the Processing of Errors and Novelty , 2012, The Journal of Neuroscience.

[11]  Clay B. Holroyd,et al.  Computational Models of Anterior Cingulate Cortex: At the Crossroads between Prediction and Effort , 2017, Front. Neurosci..

[12]  Danesh Shahnazian,et al.  Electrophysiological responses of medial prefrontal cortex to feedback at different levels of hierarchy , 2018, NeuroImage.

[13]  N. Daw.  Are we of two minds? , 2018, Nature Neuroscience.

[14]  Clay B. Holroyd,et al.  Frontal midline theta and N200 amplitude reflect complementary information about expectancy and outcome evaluation. , 2013, Psychophysiology.

[15]  L. Deserno,et al.  Devaluation and sequential decisions: linking goal-directed and model-based behavior , 2014, Front. Hum. Neurosci..

[16]  P. Dayan,et al.  States versus Rewards: Dissociable Neural Prediction Error Signals Underlying Model-Based and Model-Free Reinforcement Learning , 2010, Neuron.

[17]  Shu-Chen Li,et al.  Electrophysiological correlates reflect the integration of model-based and model-free decision information , 2017, Cognitive, affective & behavioral neuroscience.

[18]  Scott T. Grafton,et al.  Action outcomes are represented in human inferior frontoparietal cortex. , 2008, Cerebral cortex.

[19]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[20]  Amy R. Bland,et al.  Electrophysiological correlates of decision making under varying levels of uncertainty , 2011, Brain Research.

[21]  T. Harmony.  The functional significance of delta oscillations in cognitive processing , 2013, Front. Integr. Neurosci..

[22]  Timothy E. J. Behrens,et al.  Dissociable effects of surprise and model update in parietal and anterior cingulate cortex , 2013, Proceedings of the National Academy of Sciences.

[23]  C. Holroyd,et al.  Episodic, Semantic, Pavlovian, and Procedural Cognitive Maps , 2017, bioRxiv.

[24]  P. Dayan,et al.  Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control , 2005, Nature Neuroscience.

[25]  Andy J. Wills,et al.  Model-free and model-based reward prediction errors in EEG , 2018, NeuroImage.

[26]  Ernest Mas-Herrero,et al.  Frontal Theta Oscillatory Activity Is a Common Mechanism for the Computation of Unexpected Outcomes and Learning Rate , 2014, Journal of Cognitive Neuroscience.

[27]  Mattias P. Karlsson,et al.  Network Resets in Medial Prefrontal Cortex Mark the Onset of Behavioral Uncertainty , 2012, Science.

[28]  Joshua W. Brown,et al.  Medial prefrontal cortex as an action-outcome predictor , 2011, Nature Neuroscience.

[29]  John J. B. Allen,et al.  Theta lingua franca: a common mid-frontal substrate for action monitoring processes. , 2012, Psychophysiology.

[30]  Thomas H. B. FitzGerald,et al.  Disruption of Dorsolateral Prefrontal Cortex Decreases Model-Based in Favor of Model-free Control in Humans , 2013, Neuron.

[31]  T. Inouye,et al.  Medial prefrontal cortex generates frontal midline theta rhythm. , 1999, Neuroreport.

[32]  A. Owen,et al.  Planning and problem solving: From neuropsychology to functional neuroimaging , 2006, Journal of Physiology-Paris.

[33]  David J. Foster,et al.  Reverse replay of behavioural sequences in hippocampal place cells during the awake state , 2006, Nature.

[34]  Dylan A. Simon,et al.  Neural Correlates of Forward Planning in a Spatial Decision Task in Humans , 2011, The Journal of Neuroscience.

[35]  D. Hassabis,et al.  Neural Mechanisms of Hierarchical Planning in a Virtual Subway Network , 2016, Neuron.

[36]  G. Knyazev.  Motivation, emotion, and their inhibitory control mirrored in brain oscillations , 2007, Neuroscience & Biobehavioral Reviews.

[37]  Angela J. Yu,et al.  Bayesian Prediction and Evaluation in the Anterior Cingulate Cortex , 2013, The Journal of Neuroscience.

[38]  Clay B. Holroyd,et al.  Distributed representations of action sequences in anterior cingulate cortex: A recurrent neural network approach , 2017, Psychonomic Bulletin & Review.

[39]  M. Frank,et al.  Frontal theta as a mechanism for cognitive control , 2014, Trends in Cognitive Sciences.

[40]  N. Daw,et al.  The ubiquity of model-based reinforcement learning , 2012, Current Opinion in Neurobiology.

[41]  Catherine A. Hartley,et al.  From Creatures of Habit to Goal-Directed Learners , 2016, Psychological science.

[42]  P. Dayan,et al.  Model-based influences on humans’ choices and striatal prediction errors , 2011, Neuron.