The neural mechanisms of learning from competitors

Learning from competitors poses a challenge for existing theories of reward-based learning, which assume that rewarded actions are more likely to be executed in the future. Such a learning mechanism would disadvantage a player in a competitive situation because, since the competitor's loss is the player's gain, reward might become associated with an action the player should themselves avoid. Using fMRI, we investigated the neural activity of humans competing with a computer in a foraging task. We observed neural activity that represented the variables required for learning from competitors: the actions of the competitor (in the player's motor and premotor cortex) and the reward prediction error arising from the competitor's feedback. In particular, regions positively correlated with the unexpected loss of the competitor (which was beneficial to the player) included the striatum and those regions previously implicated in response inhibition. Our results suggest that learning in such contexts may involve the competitor's unexpected losses activating regions of the player's brain that subserve response inhibition, as the player learns to avoid the actions that produced them.

[1]  G. Rizzolatti The mirror neuron system and its function in humans , 2005, Anatomy and Embryology.

[2]  Timothy E. J. Behrens,et al.  Choice, uncertainty and value in prefrontal and cingulate cortex , 2008, Nature Neuroscience.

[3]  S. Cochin,et al.  Observation and execution of movement: similarities demonstrated by quantified electroencephalography , 1999, The European journal of neuroscience.

[4]  John A. Nelder,et al.  A Simplex Method for Function Minimization , 1965, Comput. J..

[5]  J. Cohen,et al.  Dissociating the role of the dorsolateral prefrontal and anterior cingulate cortex in cognitive control. , 2000, Science.

[6]  Irene Daum,et al.  Subcortical contributions to multitasking and response inhibition , 2008, Behavioural Brain Research.

[7]  M. Kahana,et al.  Human Substantia Nigra Neurons Encode Unexpected Financial Rewards , 2009, Science.

[8]  G. Rizzolatti,et al.  Neurophysiological mechanisms underlying the understanding and imitation of action , 2001, Nature Reviews Neuroscience.

[9]  Timothy Edward John Behrens,et al.  How Green Is the Grass on the Other Side? Frontopolar Cortex and the Evidence in Favor of Alternative Courses of Action , 2009, Neuron.

[10]  M. Brass,et al.  Imitation: is cognitive neuroscience solving the correspondence problem? , 2005, Trends in Cognitive Sciences.

[11]  G. Rizzolatti,et al.  Hearing Sounds, Understanding Actions: Action Representation in Mirror Neurons , 2002, Science.

[12]  Ehud Zohary,et al.  A Mirror Representation of Others' Actions in the Human Anterior Parietal Cortex , 2006, The Journal of Neuroscience.

[13]  Jonathan D. Cohen,et al.  Improved Assessment of Significant Activation in Functional Magnetic Resonance Imaging (fMRI): Use of a Cluster‐Size Threshold , 1995, Magnetic resonance in medicine.

[14]  Karl J. Friston,et al.  Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning , 2004, Science.

[15]  Jeremy R. Reynolds,et al.  Neural Mechanisms of Transient and Sustained Cognitive Control during Task Switching , 2003, Neuron.

[16]  A. Dufour,et al.  Motor cortex activation induced by a mirror: evidence from lateralized readiness potentials. , 2008, Journal of neurophysiology.

[17]  Katya Rubia,et al.  Right inferior prefrontal cortex mediates response inhibition while mesial prefrontal cortex is responsible for error detection , 2003, NeuroImage.

[18]  Beatriz Luna,et al.  Algebra and the adolescent brain , 2004, Trends in Cognitive Sciences.

[19]  M. Rushworth Intention, Choice, and the Medial Frontal Cortex , 2008, Annals of the New York Academy of Sciences.

[20]  G. Rizzolatti,et al.  The mirror neuron system. , 2009, Archives of neurology.

[21]  Samuel M. McClure,et al.  Temporal Prediction Errors in a Passive Learning Task Activate Human Striatum , 2003, Neuron.

[22]  Timothy Edward John Behrens,et al.  Triangulating a Cognitive Control Network Using Diffusion-Weighted Magnetic Resonance Imaging (MRI) and Functional MRI , 2007, The Journal of Neuroscience.

[23]  J. Wickens,et al.  A cellular mechanism of reward-related learning , 2001, Nature.

[24]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[25]  E. Koechlin,et al.  The role of the anterior prefrontal cortex in human cognition , 1999, Nature.

[26]  R. Elliott,et al.  Dissociable functions in the medial and lateral orbitofrontal cortex: evidence from human neuroimaging studies. , 2000, Cerebral cortex.

[27]  Kevin Murphy,et al.  Individual differences in the functional neuroanatomy of inhibitory control , 2006, Brain Research.

[28]  H. Leung,et al.  Common and Differential Ventrolateral Prefrontal Activity during Inhibition of Hand and Eye Movements , 2007, The Journal of Neuroscience.

[29]  J. Palacios,et al.  Dopamine receptors in human brain: Autoradiographic distribution of D1 sites , 1989, Neuroscience.

[30]  Peter Dayan,et al.  A Neural Substrate of Prediction and Reward , 1997, Science.

[31]  Russell A. Poldrack,et al.  The Cognitive Neuroscience of Response Inhibition: Relevance for Genetic Research in Attention-Deficit/Hyperactivity Disorder , 2005, Biological Psychiatry.

[32]  Riitta Hari,et al.  Actor's and observer's primary motor cortices stabilize similarly after seen or heard motor actions , 2007, Proceedings of the National Academy of Sciences.

[33]  E. Koechlin,et al.  Anterior Prefrontal Function and the Limits of Human Decision-Making , 2007, Science.

[34]  C. Summerfield,et al.  An information theoretical approach to prefrontal executive function , 2007, Trends in Cognitive Sciences.

[35]  H. Duvernoy The Human Brain , 1999, Springer Vienna.

[36]  M. Mishkin,et al.  Perseverative interference in monkeys following selective lesions of the inferior prefrontal convexity , 1970, Experimental Brain Research.

[37]  James Mark Baldwin,et al.  The Play of Animals , 1899 .

[38]  P. Hluštík,et al.  Effects of spatial smoothing on fMRI group inferences. , 2008, Magnetic resonance imaging.

[39]  H. Akaike A new look at the statistical model identification , 1974 .

[40]  A. Owen,et al.  Anterior prefrontal cortex: insights into function from anatomy and neuroimaging , 2004, Nature Reviews Neuroscience.

[41]  Scott T. Grafton,et al.  Actions or Hand-Object Interactions? Human Inferior Frontal Cortex and Action Observation , 2003, Neuron.

[42]  Michael J. Frank,et al.  Hold Your Horses: Impulsivity, Deep Brain Stimulation, and Medication in Parkinsonism , 2007, Science.

[43]  Karl J. Friston,et al.  Modelling Geometric Deformations in Epi Time Series , 2022 .

[44]  Peter Dayan,et al.  Theoretical Neuroscience: Computational and Mathematical Modeling of Neural Systems , 2001 .

[45]  G. Rizzolatti,et al.  Activation of human primary motor cortex during action observation: a neuromagnetic study. , 1998, Proceedings of the National Academy of Sciences of the United States of America.

[46]  J. Mink THE BASAL GANGLIA: FOCUSED SELECTION AND INHIBITION OF COMPETING MOTOR PROGRAMS , 1996, Progress in Neurobiology.

[47]  J. Palacios,et al.  Dopamine receptors in human brain: Autoradiographic distribution of D2 sites , 1989, Neuroscience.

[48]  Winston D. Byblow,et al.  Stop and Go: The Neural Basis of Selective Movement Prevention , 2009, Journal of Cognitive Neuroscience.

[49]  Samuel M. McClure,et al.  BOLD Responses Reflecting Dopaminergic Signals in the Human Ventral Tegmental Area , 2008, Science.

[50]  W. Schultz,et al.  Adaptive Coding of Reward Value by Dopamine Neurons , 2005, Science.

[51]  D. Barraclough,et al.  Prefrontal cortex and decision making in a mixed-strategy game , 2004, Nature Neuroscience.

[52]  Markus Ullsperger,et al.  When Errors Are Rewarding , 2009, The Journal of Neuroscience.

[53]  G. Rizzolatti,et al.  Neural Circuits Involved in the Recognition of Actions Performed by Nonconspecifics: An fMRI Study , 2004, Journal of Cognitive Neuroscience.

[54]  E. Stein,et al.  Right hemispheric dominance of inhibitory control: an event-related functional MRI study. , 1999, Proceedings of the National Academy of Sciences of the United States of America.

[55]  B. Richmond,et al.  Response differences in monkey TE and perirhinal cortex: stimulus association related to reward schedules. , 2000, Journal of neurophysiology.

[56]  Jinhu Xiong,et al.  Neuroimaging of inhibitory control areas in children with attention deficit hyperactivity disorder who were treatment naive or in long-term treatment. , 2006, The American journal of psychiatry.

[57]  Michael J. Frank,et al.  By Carrot or by Stick: Cognitive Reinforcement Learning in Parkinsonism , 2004, Science.

[58]  John N. J. Reynolds,et al.  Dopamine-dependent plasticity of corticostriatal synapses , 2002, Neural Networks.

[59]  C. Heyes Causes and consequences of imitation , 2001, Trends in Cognitive Sciences.

[60]  M. Iacoboni Neural mechanisms of imitation , 2005, Current Opinion in Neurobiology.

[61]  Y. Miyashita,et al.  No‐go dominant brain activity in human inferior prefrontal cortex revealed by functional magnetic resonance imaging , 1998, The European journal of neuroscience.

[62]  P. Dayan,et al.  Cortical substrates for exploratory decisions in humans , 2006, Nature.

[63]  Michael J. Frank,et al.  Hold your horses: A dynamic computational role for the subthalamic nucleus in decision making , 2006, Neural Networks.

[64]  E. Courchesne,et al.  Prediction and preparation, fundamental functions of the cerebellum. , 1997, Learning & memory.

[65]  Edson Oliveira,et al.  The Human Nucleus Accumbens: Where Is It? A Stereotactic, Anatomical and Magnetic Resonance Imaging Study , 2008, Neuromodulation : journal of the International Neuromodulation Society.

[66]  R. Poldrack,et al.  Cortical and Subcortical Contributions to Stop Signal Response Inhibition: Role of the Subthalamic Nucleus , 2006, The Journal of Neuroscience.

[67]  P. Dayan,et al.  A framework for mesencephalic dopamine systems based on predictive Hebbian learning , 1996, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[68]  E. Bullmore,et al.  Mapping Motor Inhibition: Conjunctive Brain Activations across Different Versions of Go/No-Go and Stop Tasks , 2001, NeuroImage.

[69]  Kevin N. Gurney,et al.  The Basal Ganglia and Cortex Implement Optimal Decision Making Between Alternative Actions , 2007, Neural Computation.

[70]  R. Elliott,et al.  Differential Neural Responses during Performance of Matching and Nonmatching to Sample Tasks at Two Delay Intervals , 1999, The Journal of Neuroscience.

[71]  H. Leung,et al.  Cortical activity during manual response inhibition guided by color and orientation cues , 2009, Brain Research.

[72]  E. Miller,et al.  An integrative theory of prefrontal cortex function. , 2001, Annual review of neuroscience.