Neuro-computational account of arbitration between imitation and emulation during human observational learning

In observational learning (OL), organisms learn from observing the behavior of others. There are at least two distinct strategies for OL. Imitation involves learning to repeat the previous actions of other agents, while in emulation, learning proceeds from inferring the goals and intentions of others. While putative neural correlates for these forms of learning have been identified, a fundamental question remains unaddressed: how does the brain decides which strategy to use in a given situation? Here we developed a novel computational model in which arbitration between the strategies is determined by the predictive reliability, such that control over behavior is adaptively weighted toward the strategy with the most reliable prediction. To test the theory, we designed a novel behavioral task in which our experimental manipulations produced dissociable effects on the reliability of the two strategies. Participants performed this task while undergoing fMRI in two independent studies (the second a pre-registered replication of the first). Behavior manifested patterns consistent with both emulation and imitation and flexibly changed between the two strategies as expected from the theory. Computational modelling revealed that behavior was best described by an arbitration model, in which the reliability of the emulation strategy determined the relative weights allocated to behavior for each strategy. Emulation reliability - the model’s arbitration signal - was encoded in the ventrolateral prefrontal cortex, temporoparietal junction and rostral cingulate cortex. Being replicated across two fMRI studies, these findings suggest a neuro-computational mechanism for allocating control between emulation and imitation during observational learning.

[1]  Carsten Allefeld,et al.  MACS – a new SPM toolbox for model assessment, comparison and selection , 2017, Journal of Neuroscience Methods.

[2]  Frank Van Overwalle,et al.  Understanding others' actions and goals by mirror and mentalizing systems: A meta-analysis , 2009, NeuroImage.

[3]  Lydia M. Hopper,et al.  Emulation, imitation, over-imitation and the scope of culture for child and chimpanzee , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[4]  Justin L. Gardner,et al.  Learning to Simulate Others' Decisions , 2012, Neuron.

[5]  B. Balleine,et al.  Human and Rodent Homologies in Action Control: Corticostriatal Determinants of Goal-Directed and Habitual Action , 2010, Neuropsychopharmacology.

[6]  Elizabeth Gilbert,et al.  Reproducibility Project: Results (Part of symposium called "The Reproducibility Project: Estimating the Reproducibility of Psychological Science") , 2014 .

[7]  Caroline Catmur,et al.  Associative sequence learning: the role of experience in the development of imitation and the mirror system , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[8]  Raymond J. Dolan,et al.  Disentangling the Roles of Approach, Activation and Valence in Instrumental and Pavlovian Responding , 2011, PLoS Comput. Biol..

[9]  Karl J. Friston,et al.  Conjunction revisited , 2005, NeuroImage.

[10]  John P O'Doherty,et al.  A causal account of the brain network computations underlying strategic social behavior , 2017, Nature Neuroscience.

[11]  P. Dayan,et al.  Model-based influences on humans’ choices and striatal prediction errors , 2011, Neuron.

[12]  Karl J. Friston,et al.  Neural Mechanisms of Belief Inference during Cooperative Games , 2010, The Journal of Neuroscience.

[13]  J. O'Doherty,et al.  The Behavioral and Neural Mechanisms Underlying the Tracking of Expertise , 2013, Neuron.

[14]  John P. O'Doherty,et al.  Human Dorsal Striatum Encodes Prediction Errors during Observational Learning of Instrumental Actions , 2012, Journal of Cognitive Neuroscience.

[15]  Jonathan W. Peirce,et al.  PsychoPy—Psychophysics software in Python , 2007, Journal of Neuroscience Methods.

[16]  Carolina Feher da Silva,et al.  Humans are primarily model-based and not model-free learners in the two-stage task , 2019, bioRxiv.

[17]  Daniel R. Lametti,et al.  Cognitive Neuroscience: The Neural Basis of Motor Learning by Observing , 2016, Current Biology.

[18]  Leif D. Nelson,et al.  False-Positive Psychology , 2011, Psychological science.

[19]  G. Rizzolatti,et al.  The mirror-neuron system. , 2004, Annual review of neuroscience.

[20]  Nathaniel D. Daw,et al.  Trial-by-trial data analysis using computational models , 2011 .

[21]  Mel W. Khaw,et al.  Normalization is a general neural mechanism for context-dependent decision making , 2013, Proceedings of the National Academy of Sciences.

[22]  M. Nielsen,et al.  Copying actions and copying outcomes: social learning through the second year. , 2006, Developmental psychology.

[23]  Richard S. J. Frackowiak,et al.  Other minds in the brain: a functional imaging study of “theory of mind” in story comprehension , 1995, Cognition.

[24]  Shinsuke Shimojo,et al.  Neural Computations Underlying Arbitration between Model-Based and Model-free Learning , 2013, Neuron.

[25]  Sang Wan Lee,et al.  Neurostimulation Reveals Context-Dependent Arbitration Between Model-Based and Model-Free Reinforcement Learning. , 2019, Cerebral cortex.

[26]  N. Tzourio-Mazoyer,et al.  Automated Anatomical Labeling of Activations in SPM Using a Macroscopic Anatomical Parcellation of the MNI MRI Single-Subject Brain , 2002, NeuroImage.

[27]  Wolfgang M Pauli,et al.  In vivo delineation of subdivisions of the human amygdaloid complex in a high‐resolution group template , 2016, Human brain mapping.

[28]  Thomas E. Nichols,et al.  Scanning the horizon: towards transparent and reproducible neuroimaging research , 2016, Nature Reviews Neuroscience.

[29]  Klaas E. Stephan,et al.  Inferring on the Intentions of Others by Hierarchical Bayesian Learning , 2014, PLoS Comput. Biol..

[30]  Jean Daunizeau,et al.  The Social Bayesian Brain: Does Mentalizing Make a Difference When We Learn? , 2014, PLoS Comput. Biol..

[31]  Brian A. Nosek,et al.  An Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[32]  Wolfgang M. Pauli,et al.  Neural computations underlying inverse reinforcement learning in the human brain , 2017, eLife.

[33]  A. Whiten,et al.  Causal knowledge and imitation/emulation switching in chimpanzees (Pan troglodytes) and children (Homo sapiens) , 2005, Animal Cognition.

[34]  Joshua Carp,et al.  On the Plurality of (Methodological) Worlds: Estimating the Analytic Flexibility of fMRI Experiments , 2012, Front. Neurosci..

[35]  P. Dayan,et al.  Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control , 2005, Nature Neuroscience.

[36]  John Duncan,et al.  The role of the right inferior frontal gyrus: inhibition and attentional control , 2010, NeuroImage.

[37]  B. Balleine,et al.  Goal-directed instrumental action: contingency and incentive learning and their cortical substrates , 1998, Neuropharmacology.

[38]  P. Dayan,et al.  States versus Rewards: Dissociable Neural Prediction Error Signals Underlying Model-Based and Model-Free Reinforcement Learning , 2010, Neuron.

[39]  John P O'Doherty,et al.  The application of computational models to social neuroscience: promises and pitfalls , 2018, Social neuroscience.

[40]  Mark W Woolrich,et al.  Associative learning of social value , 2008, Nature.

[41]  T. Robbins,et al.  Decision Making, Affect, and Learning: Attention and Performance XXIII , 2011 .

[42]  Zeb Kurth-Nelson,et al.  The modulation of savouring by prediction error and its effects on choice , 2016, eLife.

[43]  C. Heyes,et al.  Preschoolers' Behavioural Reenactment of ''Failed Attempts'': The Roles of Intention-Reading, Emulation and Mimicry. , 2006 .

[44]  Peter Bossaerts,et al.  Neural correlates of mentalizing-related computations during strategic interactions in humans , 2008, Proceedings of the National Academy of Sciences.

[45]  J. O'Doherty,et al.  Insights from the application of computational neuroimaging to social neuroscience , 2013, Current Opinion in Neurobiology.

[46]  Sang Wan Lee,et al.  Task complexity interacts with state-space uncertainty in the arbitration process between model-based and model-free reinforcement-learning at both behavioral and neural levels , 2018, bioRxiv.

[47]  C. Heyes,et al.  Testing for imitative and nonimitative social learning in the budgerigar using a two-object/two-action test , 2002, Animal Behaviour.

[48]  J. Pearce,et al.  A model for Pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli. , 1980, Psychological review.

[49]  C. Keysers,et al.  The Observation and Execution of Actions Share Motor and Somatosensory Voxels in all Tested Subjects: Single-Subject Analyses of Unsmoothed fMRI Data , 2008, Cerebral cortex.

[50]  N. Daw,et al.  Differential roles of human striatum and amygdala in associative learning , 2011, Nature Neuroscience.

[51]  Thomas E. Nichols,et al.  Best practices in data analysis and sharing in neuroimaging using MRI , 2017, Nature Neuroscience.

[52]  Andrew Y. Ng,et al.  Pharmacokinetics of a novel formulation of ivermectin after administration to goats , 2000, ICML.

[53]  C. Frith,et al.  The Neural Basis of Mentalizing , 2006, Neuron.

[54]  J. Russell,et al.  The ghost condition: imitation versus emulation in young children's observational learning. , 2004, Developmental psychology.

[55]  E. Koechlin,et al.  The Importance of Falsification in Computational Cognitive Modeling , 2017, Trends in Cognitive Sciences.

[56]  G. Rizzolatti,et al.  Premotor cortex and the recognition of motor actions. , 1996, Brain research. Cognitive brain research.

[57]  W. Schultz,et al.  Neural mechanisms of observational learning , 2010, Proceedings of the National Academy of Sciences.

[58]  C. Heyes,et al.  Mirror neurons: from origin to function. , 2014, The Behavioral and brain sciences.

[59]  John-Dylan Haynes,et al.  How to avoid mismodelling in GLM-based fMRI data analysis: cross-validated Bayesian model selection , 2016, NeuroImage.

[60]  S. Eickhoff,et al.  Neuroscience and Biobehavioral Reviews Three Key Regions for Supervisory Attentional Control: Evidence from Neuroimaging Meta-analyses , 2022 .