Interactions between frontal and posterior oscillatory dynamics support adjustment of stimulus processing during reinforcement learning

ABSTRACT Reinforcement learning (RL) in humans is subserved by a network of striatal and frontal brain areas. The electrophysiological signatures of feedback evaluation are increasingly well understood, but how those signatures relate to the use of feedback to guide subsequent behavioral adjustment remains unclear. One mechanism for post‐feedback behavioral optimization is the modulation of sensory processing. We used source‐reconstructed MEG to test whether feedback affects the interactions between sources of oscillatory activity in the learning network and task‐relevant stimulus‐processing areas. Participants performed a probabilistic RL task in which they learned associations between colored faces and response buttons using trial‐and‐error feedback. Delta‐band (2–4Hz) and theta‐band (4–8Hz) power in multiple frontal regions were sensitive to feedback valence. Low and high beta‐band power (12–20 and 20–30Hz) in occipital, parietal, and temporal regions differentiated between color and face information. Consistent with our hypothesis, single‐trial power‐power correlations between frontal and posterior‐sensory areas were modulated by the interaction between feedback valence and the relevant stimulus characteristic (color versus identity). These results suggest that long‐range oscillatory coupling supports post‐feedback updating of stimulus processing. HIGHLIGHTSFrontal delta and theta power reflect feedback valence.Posterior low and high beta power reflect task‐relevant stimulus information.Only frontal‐posterior power‐power correlations show interactions of both aspects.Post‐feedback long‐range connectivity may underlie updating of stimulus processing.

[1]  P. Uhlhaas,et al.  Working memory and neural oscillations: alpha–gamma versus theta–gamma codes for distinct WM information? , 2014, Trends in Cognitive Sciences.

[2]  G. Knyazev,et al.  Neuroscience and Biobehavioral Reviews , 2012 .

[3]  Manuel Schabus,et al.  Fronto-parietal EEG coherence in theta and upper alpha reflect central executive functions of working memory. , 2005, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[4]  B. W. van Dijk,et al.  Feasibility of clinical Magnetoencephalography (MEG) functional mapping in the presence of dental artefacts , 2013, Clinical Neurophysiology.

[5]  O. Jensen,et al.  Prestimulus alpha and mu activity predicts failure to inhibit motor responses , 2009, Human brain mapping.

[6]  Matthew J. Brookes,et al.  Optimising experimental design for MEG beamformer imaging , 2008, NeuroImage.

[7]  Timothy Edward John Behrens,et al.  How Green Is the Grass on the Other Side? Frontopolar Cortex and the Evidence in Favor of Alternative Courses of Action , 2009, Neuron.

[8]  Caroline Di Bernardi Luft,et al.  Learning from feedback: The neural mechanisms of feedback processing facilitating better performance , 2014, Behavioural Brain Research.

[9]  John J. B. Allen,et al.  Theta lingua franca: a common mid-frontal substrate for action monitoring processes. , 2012, Psychophysiology.

[10]  Satu Palva,et al.  Cross-frequency synchronization connects networks of fast and slow oscillations during visual working memory maintenance , 2016, eLife.

[11]  J. Palva,et al.  Neuronal synchrony reveals working memory networks and predicts individual memory capacity , 2010, Proceedings of the National Academy of Sciences.

[12]  J. Bhattacharya,et al.  High-Learners Present Larger Mid-Frontal Theta Power and Connectivity in Response to Incorrect Performance Feedback , 2013, The Journal of Neuroscience.

[13]  K. R. Ridderinkhof,et al.  Not All Errors Are Alike: Theta and Alpha EEG Dynamics Relate to Differences in Error-Processing Dynamics , 2012, The Journal of Neuroscience.

[14]  T. Sejnowski,et al.  Synchrony of Thalamocortical Inputs Maximizes Cortical Reliability , 2010, Science.

[15]  Joerg F. Hipp,et al.  Accounting for Linear Transformations of EEG and MEG Data in Source Analysis , 2015, PloS one.

[16]  Rodrigo F. Salazar,et al.  Content-Specific Fronto-Parietal Synchronization During Visual Working Memory , 2012, Science.

[17]  Ernest Mas-Herrero,et al.  Frontal Theta Oscillatory Activity Is a Common Mechanism for the Computation of Unexpected Outcomes and Learning Rate , 2014, Journal of Cognitive Neuroscience.

[18]  R. Desimone,et al.  High-Frequency, Long-Range Coupling Between Prefrontal and Visual Cortex During Attention , 2009, Science.

[19]  Monica Fabiani,et al.  Validation of a method for coregistering scalp recording locations with 3D structural MR images , 2008, Human brain mapping.

[20]  M. Siegel,et al.  A framework for local cortical oscillation patterns , 2011, Trends in Cognitive Sciences.

[21]  W. Singer,et al.  Modulation of Neuronal Interactions Through Neuronal Synchronization , 2007, Science.

[22]  G. R. Barnes,et al.  A Quantitative Assessment of the Sensitivity of Whole-Head MEG to Activity in the Adult Human Cortex , 2002, NeuroImage.

[23]  Robert T. Knight,et al.  Five-dimensional neuroimaging: Localization of the time–frequency dynamics of cortical activity , 2008, NeuroImage.

[24]  G. Nolte The magnetic lead field theorem in the quasi-static approximation and its use for magnetoencephalography forward calculation in realistic volume conductors. , 2003, Physics in medicine and biology.

[25]  P. Fries A mechanism for cognitive dynamics: neuronal communication through neuronal coherence , 2005, Trends in Cognitive Sciences.

[26]  Dimitri M. Kullmann,et al.  Oscillations and Filtering Networks Support Flexible Routing of Information , 2010, Neuron.

[27]  Michael X. Cohen,et al.  Theta Dynamics Reveal Domain-specific Control over Stimulus and Response Conflict , 2012, Journal of Cognitive Neuroscience.

[28]  T. Maia Reinforcement learning, conditioning, and the brain: Successes and challenges , 2009, Cognitive, affective & behavioral neuroscience.

[29]  N. Cashdollar,et al.  Hippocampus-dependent and -independent theta-networks of active maintenance , 2009, Proceedings of the National Academy of Sciences.

[30]  G. R. Barnes,et al.  Elsevier Editorial System(tm) for Journal of Neuroscience Methods Manuscript Draft Manuscript Number: JNEUMETH-D-07-00538R1 Title: Effective electromagnetic noise cancellation with beamformers and synthetic gradiometry in shielded and partly-shielded environments , 2008 .

[31]  P. Dayan,et al.  Cortical substrates for exploratory decisions in humans , 2006, Nature.

[32]  R. Dolan,et al.  Cholinergic Enhancement of Visual Attention and Neural Oscillations in the Human Brain , 2012, Current Biology.

[33]  R. Oostenveld,et al.  Nonparametric statistical testing of EEG- and MEG-data , 2007, Journal of Neuroscience Methods.

[34]  Michael X. Cohen,et al.  Reward expectation modulates feedback-related negativity and EEG spectra , 2007, NeuroImage.

[35]  Michael X. Cohen,et al.  Frontal Oscillatory Dynamics Predict Feedback Learning and Action Adjustment , 2011, Journal of Cognitive Neuroscience.

[36]  K Sekihara,et al.  NUTMEG: a neuromagnetic source reconstruction toolbox. , 2004, Neurology & clinical neurophysiology : NCN.

[37]  P. Fries Rhythms for Cognition: Communication through Coherence , 2015, Neuron.

[38]  Michael X Cohen,et al.  Analyzing Neural Time Series Data: Theory and Practice , 2014 .

[39]  Michael X. Cohen,et al.  Frontal theta reflects uncertainty and unexpectedness during exploration and exploitation. , 2012, Cerebral cortex.

[40]  Scott D Slotnick,et al.  Memory for color reactivates color processing region , 2009, Neuroreport.

[41]  A. Engel,et al.  Spectral fingerprints of large-scale neuronal interactions , 2012, Nature Reviews Neuroscience.

[42]  K. R. Ridderinkhof,et al.  Unconscious Errors Enhance Prefrontal-Occipital Oscillatory Synchrony , 2009, Front. Hum. Neurosci..

[43]  R. Eckhorn,et al.  Amplitude envelope correlation detects coupling among incoherent brain signals , 2000, Neuroreport.

[44]  A. Takahashi,et al.  Anterior striatum with dysmorphic neurons associated with the epileptogenesis of focal cortical dysplasia , 2010, Seizure.

[45]  Michael X. Cohen,et al.  Cortical electrophysiological network dynamics of feedback learning , 2011, Trends in Cognitive Sciences.

[46]  Robert Oostenveld,et al.  FieldTrip: Open Source Software for Advanced Analysis of MEG, EEG, and Invasive Electrophysiological Data , 2010, Comput. Intell. Neurosci..

[47]  Lars-Göran Nilsson,et al.  Reactivation of Motor Brain Areas during Explicit Memory for Actions , 2001, NeuroImage.

[48]  Jane R. Garrison,et al.  Prediction error in reinforcement learning: A meta-analysis of neuroimaging studies , 2013, Neuroscience & Biobehavioral Reviews.

[49]  R. Nigbur,et al.  Theta power as a marker for cognitive interference , 2011, Clinical Neurophysiology.

[50]  J. Lisman,et al.  The Theta-Gamma Neural Code , 2013, Neuron.

[51]  Katsuyuki Sakai,et al.  Prefrontal Set Activity Predicts Rule-Specific Neural Processing during Subsequent Cognitive Performance , 2006, The Journal of Neuroscience.

[52]  Riitta Hari,et al.  Removal of magnetoencephalographic artifacts with temporal signal‐space separation: Demonstration with single‐trial auditory‐evoked responses , 2009, Human brain mapping.

[53]  K. R. Ridderinkhof,et al.  EEG Source Reconstruction Reveals Frontal-Parietal Dynamics of Spatial Conflict Processing , 2013, PloS one.

[54]  Karl J. Friston,et al.  Behavioral / Systems / Cognitive Striatal Prediction Error Modulates Cortical Coupling , 2010 .

[55]  M. Heil,et al.  Distinct Cortical Activation Patterns during Long-Term Memory Retrieval of Verbal, Spatial, and Color Information , 1995, Journal of Cognitive Neuroscience.

[56]  G. V. Simpson,et al.  Anticipatory Biasing of Visuospatial Attention Indexed by Retinotopically Specific α-Bank Electroencephalography Increases over Occipital Cortex , 2000, The Journal of Neuroscience.

[57]  Mircea Ariel Schoenfeld,et al.  Magneto- and electroencephalographic manifestations of reward anticipation and delivery , 2012, NeuroImage.

[58]  Pascal Fries,et al.  Communication through coherence with inter-areal delays , 2015, Current Opinion in Neurobiology.

[59]  Arnaud Delorme,et al.  EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis , 2004, Journal of Neuroscience Methods.

[60]  Marcel van Gerven,et al.  Measuring directionality between neuronal oscillations of different frequencies , 2015, NeuroImage.

[61]  E. Tulving,et al.  Reactivation of encoding-related brain activity during memory retrieval. , 2000, Proceedings of the National Academy of Sciences of the United States of America.

[62]  S. Haber,et al.  The Reward Circuit: Linking Primate Anatomy and Human Imaging , 2010, Neuropsychopharmacology.

[63]  L. Hoadley A DEVICE FOR MICROMANIPULATION. , 1934, Science.

[64]  Cornelis J. Stam,et al.  Changes in resting-state directed connectivity in cortico-subcortical networks correlate with cognitive function in Parkinson’s disease , 2017, Clinical Neurophysiology.

[65]  Marco Rotonda,et al.  EEG oscillatory activity associated to monetary gain and loss signals in a learning task: effects of attentional impulsivity and learning ability. , 2012, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[66]  Joachim Gross,et al.  Good practice for conducting and reporting MEG research , 2013, NeuroImage.

[67]  N. Yeung,et al.  Reward Activates Stimulus-Specific and Task-Dependent Representations in Visual Association Cortices , 2014, The Journal of Neuroscience.

[68]  Benjamin A. E. Hunt,et al.  Measuring electrophysiological connectivity by power envelope correlation: a technical review on MEG methods , 2015, Physics in medicine and biology.

[69]  Julia H. Keller,et al.  A quantitative assessment of the sensitivity of the downstream midlatitude flow response to extratropical transition of tropical cyclones , 2015 .

[70]  Jan R Wessel,et al.  Unexpected Events Induce Motor Slowing via a Brain Mechanism for Action-Stopping with Global Suppressive Effects , 2013, The Journal of Neuroscience.

[71]  Zoe Kourtzi,et al.  Decoding the future from past experience: learning shapes predictions in early visual cortex , 2015, Journal of neurophysiology.

[72]  Joydeep Bhattacharya,et al.  Processing Graded Feedback: Electrophysiological Correlates of Learning from Small and Large Errors , 2014, Journal of Cognitive Neuroscience.

[73]  M. Packard,et al.  Changes in corticostriatal connectivity during reinforcement learning in humans , 2015, Human brain mapping.

[74]  H. Seo,et al.  Neural basis of reinforcement learning and decision making. , 2012, Annual review of neuroscience.

[75]  A. Rodríguez-Fornells,et al.  The role of high-frequency oscillatory activity in reward processing and learning , 2015, Neuroscience & Biobehavioral Reviews.

[76]  Michael X. Cohen,et al.  Dynamic Interactions between Large-Scale Brain Networks Predict Behavioral Adaptation after Perceptual Errors , 2012, Cerebral cortex.

[77]  K. R. Ridderinkhof,et al.  Aging affects medial but not anterior frontal learning-related theta oscillations , 2014, Neurobiology of Aging.

[78]  R. Knight,et al.  The functional role of cross-frequency coupling , 2010, Trends in Cognitive Sciences.

[79]  W. Singer,et al.  Dynamic predictions: Oscillations and synchrony in top–down processing , 2001, Nature Reviews Neuroscience.

[80]  James F. Cavanagh,et al.  Frontal theta links prediction errors to behavioral adaptation in reinforcement learning , 2010, NeuroImage.

[81]  Timothy E. J. Behrens,et al.  Review Frontal Cortex and Reward-guided Learning and Decision-making Figure 1. Frontal Brain Regions in the Macaque Involved in Reward-guided Learning and Decision-making Finer Grained Anatomical Divisions with Frontal Cortical Systems for Reward-guided Behavior , 2022 .

[82]  Natalie C. Ebner,et al.  Age of face matters: Age-group differences in ratings of young and old faces , 2008, Behavior research methods.

[83]  S. Taulu,et al.  Spatiotemporal signal space separation method for rejecting nearby interference in MEG measurements , 2006, Physics in medicine and biology.

[84]  Juliana Yordanova,et al.  Aging and error processing: Time-frequency analysis of error-related potentials. , 2005 .

[85]  Roland R. Lee,et al.  Resting-State Magnetoencephalography Reveals Different Patterns of Aberrant Functional Connectivity in Combat-Related Mild Traumatic Brain Injury. , 2017, Journal of neurotrauma.

[86]  W Singer,et al.  Visual feature integration and the temporal correlation hypothesis. , 1995, Annual review of neuroscience.

[87]  Juliana Yordanova,et al.  Error-Related Oscillations , 2009 .

[88]  Seppo P. Ahlfors,et al.  Sensitivity of MEG and EEG to Source Orientation , 2010, Brain Topography.