Neural Computations Underlying Causal Structure Learning

Behavioral evidence suggests that beliefs about causal structure constrain associative learning, determining which stimuli can enter into association, as well as the functional form of that association. Bayesian learning theory provides one mechanism by which structural beliefs can be acquired from experience, but the neural basis of this mechanism is poorly understood. We studied this question with a combination of behavioral, computational, and neuroimaging techniques. Male and female human subjects learned to predict an outcome based on cue and context stimuli while being scanned using fMRI. Using a model-based analysis of the fMRI data, we show that structure learning signals are encoded in posterior parietal cortex, lateral prefrontal cortex, and the frontal pole. These structure learning signals are distinct from associative learning signals. Moreover, representational similarity analysis and information mapping revealed that the multivariate patterns of activity in posterior parietal cortex and anterior insula encode the full posterior distribution over causal structures. Variability in the encoding of the posterior across subjects predicted variability in their subsequent behavioral performance. These results provide evidence for a neural architecture in which structure learning guides the formation of associations. SIGNIFICANCE STATEMENT Animals are able to infer the hidden structure behind causal relations between stimuli in the environment, allowing them to generalize this knowledge to stimuli they have never experienced before. A recently published computational model based on this idea provided a parsimonious account of a wide range of phenomena reported in the animal learning literature, suggesting a dedicated neural mechanism for learning this hidden structure. Here, we validate this model by measuring brain activity during a task that involves both structure learning and associative learning. We show that a distinct network of regions supports structure learning and that the neural signal corresponding to beliefs about structure predicts future behavioral performance.

[1]  Jonathan W. Pillow,et al.  A Bayesian method for reducing bias in neural representational similarity analysis , 2016, bioRxiv.

[2]  Jan Gläscher,et al.  Visualization of Group Inference Data in Functional Neuroimaging , 2009, Neuroinformatics.

[3]  Denis Cousineau,et al.  Confidence intervals in within-subject designs: A simpler solution to Loftus and Masson's method , 2005 .

[4]  Robert C. Wilson,et al.  Reinforcement Learning in Multidimensional Environments Relies on Attention Mechanisms , 2015, The Journal of Neuroscience.

[5]  R. Rescorla,et al.  Role of context in autoshaping. , 1984 .

[6]  J. Kruschke Bayesian approaches to associative learning: From passive to active learning , 2008, Learning & behavior.

[7]  Russell A. Poldrack,et al.  Orthogonalization of Regressors in fMRI Models , 2015, PloS one.

[8]  Anne G E Collins,et al.  Cognitive control over learning: creating, clustering, and generalizing task-set structure. , 2013, Psychological review.

[9]  N. Mackintosh,et al.  Context specificity of conditioning, extinction, and latent inhibition. , 1984 .

[10]  Peter Dayan,et al.  Explaining Away in Weight Space , 2000, NIPS.

[11]  R. Rescorla,et al.  A theory of Pavlovian conditioning : Variations in the effectiveness of reinforcement and nonreinforcement , 1972 .

[12]  Jonathan W. Peirce,et al.  PsychoPy—Psychophysics software in Python , 2007, Journal of Neuroscience Methods.

[13]  Steen Moeller,et al.  Multiband multislice GE‐EPI at 7 tesla, with 16‐fold acceleration using partial parallel imaging with application to high spatial and temporal whole‐brain fMRI , 2010, Magnetic resonance in medicine.

[14]  Mark E. Bouton,et al.  Context effects on conditioning, extinction, and reinstatement in an appetitive conditioning preparation , 1989 .

[15]  Timothy E. J. Behrens,et al.  Brain Systems for Probabilistic and Dynamic Prediction: Computational Specificity and Integration , 2013, PLoS biology.

[16]  Anders M. Dale,et al.  An automated labeling system for subdividing the human cerebral cortex on MRI scans into gyral based regions of interest , 2006, NeuroImage.

[17]  Fabian A. Soto,et al.  Explaining compound generalization in associative and causal learning through rational principles of dimensional generalization. , 2014, Psychological review.

[18]  Justin L. Vincent,et al.  Evidence for a frontoparietal control system revealed by intrinsic functional connectivity. , 2008, Journal of neurophysiology.

[19]  A. Pouget,et al.  Probabilistic brains: knowns and unknowns , 2013, Nature Neuroscience.

[20]  Karl J. Friston,et al.  Comparing the similarity and spatial structure of neural representations: A pattern-component model , 2011, NeuroImage.

[21]  Edmund T. Rolls,et al.  Implementation of a new parcellation of the orbitofrontal cortex in the automated anatomical labeling atlas , 2015, NeuroImage.

[22]  S. Kakade,et al.  Acquisition and extinction in autoshaping. , 2002, Psychological review.

[23]  M. Seghier The Angular Gyrus , 2013, The Neuroscientist : a review journal bringing neurobiology, neurology and psychiatry.

[24]  André J. W. van der Kouwe,et al.  Brain morphometry with multiecho MPRAGE , 2008, NeuroImage.

[25]  Simon B. Eickhoff,et al.  A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data , 2005, NeuroImage.

[26]  P. Dayan,et al.  States versus Rewards: Dissociable Neural Prediction Error Signals Underlying Model-Based and Model-Free Reinforcement Learning , 2010, Neuron.

[27]  Rainer Goebel,et al.  Information-based functional brain mapping. , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[28]  Ralph R. Miller,et al.  Assessment of the Rescorla-Wagner model. , 1995 .

[29]  József Fiser,et al.  Perceptual Decision-Making as Probabilistic Inference by Neural Sampling , 2014, Neuron.

[30]  S. Gershman Context-dependent learning and causal structure , 2017, Psychonomic bulletin & review.

[31]  N. Daw,et al.  Rethinking Extinction , 2015, Neuron.

[32]  M. D’Esposito,et al.  Frontal Cortex and the Discovery of Abstract Action Rules , 2010, Neuron.

[33]  M. Frank,et al.  Neural signature of hierarchically structured expectations predicts clustering and transfer of rule sets in reinforcement learning , 2016, Cognition.

[34]  Karl J. Friston,et al.  Bayesian model selection for group studies — Revisited , 2014, NeuroImage.

[35]  P. Dayan,et al.  Cortical substrates for exploratory decisions in humans , 2006, Nature.

[36]  Robert C. Wilson,et al.  Orbitofrontal Cortex as a Cognitive Map of Task Space , 2014, Neuron.

[37]  R. Bolles,et al.  Contextual control of the extinction of conditioned fear , 1979 .

[38]  N. Daw,et al.  Human Reinforcement Learning Subdivides Structured Action Spaces by Learning Effector-Specific Values , 2009, The Journal of Neuroscience.

[39]  Francisco Pereira,et al.  Information mapping with pattern classifiers: A comparative study , 2011, NeuroImage.

[40]  Michael J Frank,et al.  Human EEG Uncovers Latent Generalizable Rule Structure during Learning , 2014, The Journal of Neuroscience.

[41]  David Badre,et al.  Functional Magnetic Resonance Imaging Evidence for a Hierarchical Organization of the Prefrontal Cortex , 2007, Journal of Cognitive Neuroscience.

[42]  Steen Moeller,et al.  Evaluation of slice accelerations using multiband echo planar imaging at 3T , 2013, NeuroImage.

[43]  Nikolaus Kriegeskorte,et al.  Mind the drift - improving sensitivity to fMRI pattern information by accounting for temporal pattern drift , 2015, bioRxiv.

[44]  M. Frank,et al.  Mechanisms of hierarchical reinforcement learning in corticostriatal circuits 1: computational analysis. , 2012, Cerebral cortex.

[45]  N. Mackintosh,et al.  Contextual Conditional Discriminations , 1986 .

[46]  József Fiser,et al.  Neural Variability and Sampling-Based Probabilistic Representations in the Visual Cortex , 2016, Neuron.

[47]  Konrad Paul Kording,et al.  Causal Inference in Multisensory Perception , 2007, PloS one.

[48]  Joshua B Tenenbaum,et al.  Toward the neural implementation of structure learning , 2016, Current Opinion in Neurobiology.

[49]  Nikolaus Kriegeskorte,et al.  Frontiers in Systems Neuroscience Systems Neuroscience , 2022 .

[50]  Yael Niv,et al.  A Probability Distribution over Latent Causes, in the Orbitofrontal Cortex , 2016, The Journal of Neuroscience.

[51]  D. A. King,et al.  Contextual control of the extinction of conditioned fear: tests for the associative value of the context. , 1983, Journal of experimental psychology. Animal behavior processes.

[52]  Ralph R. Miller,et al.  Context as an occasion setter following either CS acquisition and extinction or CS acquisition alone , 1990 .

[53]  C. Summerfield,et al.  An information theoretical approach to prefrontal executive function , 2007, Trends in Cognitive Sciences.

[54]  Nicolas W. Schuck,et al.  Human Orbitofrontal Cortex Represents a Cognitive Map of State Space , 2016, Neuron.

[55]  Etienne Koechlin,et al.  Foundations of human reasoning in the prefrontal cortex , 2014, Science.

[56]  Y. Niv,et al.  Discovering latent causes in reinforcement learning , 2015, Current Opinion in Behavioral Sciences.

[57]  Samuel J. Gershman,et al.  A Tutorial on Bayesian Nonparametric Models , 2011, 1106.2697.

[58]  Samuel Gershman,et al.  A Unifying Probabilistic View of Associative Learning , 2015, PLoS Comput. Biol..

[59]  Stephen M. Smith,et al.  Multiplexed Echo Planar Imaging for Sub-Second Whole Brain FMRI and Fast Diffusion Imaging , 2010, PloS one.

[60]  Björn Meder,et al.  Structure induction in diagnostic causal reasoning. , 2014, Psychological review.

[61]  D. Swartzentruber,et al.  Modulatory mechanisms in Pavlovian conditioning , 1995 .

[62]  J. Tenenbaum,et al.  Structure and strength in causal induction , 2005, Cognitive Psychology.

[63]  E. Koechlin,et al.  The Architecture of Cognitive Control in the Human Prefrontal Cortex , 2003, Science.

[64]  Robert A. Legenstein,et al.  Ensembles of Spiking Neurons with Noise Support Optimal Probabilistic Inference in a Dynamically Changing Environment , 2014, PLoS Comput. Biol..

[65]  N. Mackintosh A Theory of Attention: Variations in the Associability of Stimuli with Reinforcement , 1975 .

[66]  M. Botvinick,et al.  Neural representations of events arise from temporal community structure , 2013, Nature Neuroscience.

[67]  Edgar A Ycu,et al.  Evaluation of ambiguous associations in the amygdala by learning the structure of the environment , 2016, Nature Neuroscience.

[68]  Timothy Edward John Behrens,et al.  How Green Is the Grass on the Other Side? Frontopolar Cortex and the Evidence in Favor of Alternative Courses of Action , 2009, Neuron.

[69]  N. Mackintosh,et al.  Context Specificity of Conditioning and Latent Inhibition: Evidence for a Dissociation of Latent Inhibition and Associative Interference , 1987 .

[70]  M. Bouton,et al.  Analysis of the associative and occasion-setting properties of contexts participating in a Pavlovian discrimination. , 1986 .

[71]  Alexander Borst,et al.  How does Nature Program Neuron Types? , 2008, Front. Neurosci..

[72]  N. Tzourio-Mazoyer,et al.  Automated Anatomical Labeling of Activations in SPM Using a Macroscopic Anatomical Parcellation of the MNI MRI Single-Subject Brain , 2002, NeuroImage.

[73]  J. Pearce,et al.  Theories of associative learning in animals. , 2001, Annual review of psychology.

[74]  Ralph R. Miller,et al.  Contextual potentiation of acquired behavior after devaluing direct context-US associations , 1981 .

[75]  F. J. Odling-Smee The Overshadowing of Background Stimuli: Some Effects of Varying Amounts of Training and UCS Intensity , 1978, The Quarterly journal of experimental psychology.