Reinforcement Learning and Bayesian Inference Provide Complementary Models for the Unique Advantage of Adolescents in Stochastic Reversal

During adolescence, youth venture out, explore the wider world, and are challenged to learn how to navigate novel and uncertain environments. We investigated whether adolescents are uniquely adapted to this transition, compared to younger children and adults. In a stochastic, volatile learning task with a sample of 291 participants aged 8-30, we found that adolescents 13-15 years old outperformed both younger and older participants. We developed two independent cognitive models, and used hierarchical Bayesian model fitting to assess developmental changes in underlying cognitive mechanisms. Choice parameters in both models improved monotonously. By contrast, up-date parameters peaked closest to optimal values in 13-15 year-olds. Combining both models using principal component analysis yielded new insights, revealing that three components contributed to the early to mid-adolescent performance peak. This research highlights early to mid-adolescence as a neurodevelopmental window that may be more optimal for behavioral adjustment in volatile and uncertain environments. It also shows how detailed insights can be gleaned by combining cognitive models.

[1]  M. Hennessy,et al.  A Biosocial-Affect Model of Adolescent Sensation Seeking: The Role of Affect Evaluation and Peer-Group Influence in Adolescent Drug Use , 2007, Prevention Science.

[2]  J. Willing,et al.  Pubertal onset as a critical transition for neural development and cognition , 2017, Brain Research.

[3]  Willem E. Frankenhuis,et al.  Modeling the evolution of sensitive periods , 2019, Developmental Cognitive Neuroscience.

[4]  Linda Wilbrecht,et al.  Neuroscience: Sex Hormones at Work in the Neocortex , 2019, Current Biology.

[5]  Sang Wan Lee,et al.  The structure of reinforcement-learning mechanisms in the human brain , 2015, Current Opinion in Behavioral Sciences.

[6]  B. Casey,et al.  Adjusting behavior to changing environmental demands with development , 2013, Neuroscience & Biobehavioral Reviews.

[7]  Anne G E Collins,et al.  How much of reinforcement learning is working memory, not reinforcement learning? A behavioral, computational, and neurogenetic analysis , 2012, The European journal of neuroscience.

[8]  Michael X. Cohen,et al.  Striatum-medial prefrontal cortex connectivity predicts developmental changes in reinforcement learning. , 2012, Cerebral cortex.

[9]  Linda Wilbrecht,et al.  Juvenile mice show greater flexibility in multiple choice reversal learning than adults , 2011, Developmental Cognitive Neuroscience.

[10]  K. Norman,et al.  Reinstated episodic context guides sampling-based decisions for reward , 2017, Nature Neuroscience.

[11]  Amir Homayoun Javadi,et al.  Adolescents Adapt More Slowly than Adults to Varying Reward Contingencies , 2014, Journal of Cognitive Neuroscience.

[12]  P. Dayan,et al.  Model-based influences on humans’ choices and striatal prediction errors , 2011, Neuron.

[13]  M. Frank,et al.  Anatomy of a decision: striato-orbitofrontal interactions in reinforcement learning, decision making, and reversal. , 2006, Psychological review.

[14]  Paul W. Frankland,et al.  Age-dependent changes in spatial memory retention and flexibility in mice , 2017, Neurobiology of Learning and Memory.

[15]  Laurence Steinberg,et al.  Age differences in future orientation and delay discounting. , 2009, Child development.

[16]  E. Tucker-Drob,et al.  Individual differences in the development of sensation seeking and impulsivity during adolescence: further evidence for a dual systems model. , 2011, Developmental psychology.

[17]  M. Lee How cognitive modeling can benefit from hierarchical Bayesian models. , 2011 .

[18]  W. van den Bos,et al.  Pubertal testosterone correlates with adolescent impatience and dorsal striatal activity , 2019, Developmental Cognitive Neuroscience.

[19]  A. Toga,et al.  Mapping brain maturation , 2006, Trends in Neurosciences.

[20]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[21]  Maria K. Eckstein,et al.  Distentangling the systems contributing to changes in learning during adolescence , 2019, Developmental Cognitive Neuroscience.

[22]  Robert C. Wilson,et al.  Ten simple rules for the computational modeling of behavioral data , 2019, eLife.

[23]  J. Tenenbaum,et al.  A tutorial introduction to Bayesian models of cognitive development , 2011, Cognition.

[24]  L. Wilbrecht,et al.  Transient stimulation of distinct subpopulations of striatal neurons mimics changes in action value , 2012, Nature Neuroscience.

[25]  M. D’Esposito,et al.  Frontal Cortex and the Discovery of Abstract Action Rules , 2010, Neuron.

[26]  M. Nassar,et al.  Computational neuroscience across the lifespan: Promises and pitfalls , 2017, Developmental Cognitive Neuroscience.

[27]  Thomas F. Nugent,et al.  Dynamic mapping of human cortical development during childhood through early adulthood. , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[28]  Nicholas B. Allen,et al.  Importance of investing in adolescence from a developmental science perspective , 2018, Nature.

[29]  Alan C. Evans,et al.  Brain development during childhood and adolescence: a longitudinal MRI study , 1999, Nature Neuroscience.

[30]  E. Crone,et al.  Distinct linear and non-linear trajectories of reward and punishment reversal learning during development: Relevance for dopamine's role in adolescent decision making , 2011, Developmental Cognitive Neuroscience.

[31]  Daniel Brandeis,et al.  Cognitive flexibility in adolescence: Neural and behavioral mechanisms of reward prediction error processing in adaptive decision making during development , 2015, NeuroImage.

[32]  Michael J. Frank,et al.  By Carrot or by Stick: Cognitive Reinforcement Learning in Parkinsonism , 2004, Science.

[33]  H. Kraemer,et al.  How can we learn about developmental processes from cross-sectional studies, or can we? , 2000, The American journal of psychiatry.

[34]  Timothy E. J. Behrens,et al.  Counterfactual Choice and Learning in a Neural Network Centered on Human Lateral Frontopolar Cortex , 2011, PLoS biology.

[35]  Laurence Steinberg,et al.  The Teenage Brain , 2013 .

[36]  Y. Niv Reinforcement learning in the brain , 2009 .

[37]  Sumio Watanabe,et al.  A widely applicable Bayesian information criterion , 2012, J. Mach. Learn. Res..

[38]  Juliet Y. Davidow,et al.  An Upside to Reward Sensitivity: The Hippocampus Supports Enhanced Reinforcement Learning in Adolescence , 2016, Neuron.

[39]  John Salvatier,et al.  Probabilistic programming in Python using PyMC3 , 2016, PeerJ Comput. Sci..

[40]  A. Petersen,et al.  A self-report measure of pubertal status: Reliability, validity, and initial norms , 1988, Journal of youth and adolescence.

[41]  Maria K. Eckstein,et al.  Learning under uncertainty changes during adolescence , 2020, CogSci.

[42]  Laurence Steinberg,et al.  The influence of neuroscience on US Supreme Court decisions about adolescents' criminal culpability , 2013, Nature Reviews Neuroscience.

[43]  H. Sercombe,et al.  Risk, adaptation and the functional teenage brain , 2014, Brain and Cognition.

[44]  J. Willing,et al.  Synaptic number changes in the medial prefrontal cortex across adolescence in male and female rats: A role for pubertal onset , 2016, Synapse.

[45]  Josiah R. Boivin,et al.  Does puberty mark a transition in sensitive periods for plasticity in the associative neocortex? , 2017, Brain Research.

[46]  Laurence Steinberg,et al.  Peer Influences on Adolescent Decision Making. , 2013, Current directions in psychological science.

[47]  Kentaro Katahira,et al.  How hierarchical models improve point estimates of model parameters at the individual level , 2016 .

[48]  Simo Srkk,et al.  Bayesian Filtering and Smoothing , 2013 .

[49]  栁下 祥 A critical time window for dopamine actions on the structural plasticity of dendritic spines , 2016 .

[50]  D. Bates,et al.  Fitting Linear Mixed-Effects Models Using lme4 , 2014, 1406.5823.

[51]  Stefano Palminteri,et al.  The Computational Development of Reinforcement Learning during Adolescence , 2016, PLoS Comput. Biol..

[52]  Erin Kendall Braun,et al.  Episodic Memory Encoding Interferes with Reward Learning and Decreases Striatal Prediction Errors , 2014, The Journal of Neuroscience.

[53]  Zeb Kurth-Nelson,et al.  A distributional code for value in dopamine-based reinforcement learning , 2020, Nature.

[54]  E. Koechlin,et al.  The Importance of Falsification in Computational Cognitive Modeling , 2017, Trends in Cognitive Sciences.

[55]  Adriana Galván,et al.  Frontostriatal development and probabilistic reinforcement learning during adolescence , 2017, Neurobiology of Learning and Memory.

[56]  C. D. De Dreu,et al.  The development of creative cognition across adolescence: distinct trajectories for insight and divergent thinking. , 2013, Developmental science.

[57]  H. Seo,et al.  Neural basis of reinforcement learning and decision making. , 2012, Annual review of neuroscience.

[58]  Suzanne E. Welcome,et al.  Mapping cortical change across the human life span , 2003, Nature Neuroscience.

[59]  Josiah R. Boivin,et al.  Ovarian Hormones Organize the Maturation of Inhibitory Neurotransmission in the Frontal Cortex at Puberty Onset in Female Mice , 2017, Current Biology.

[60]  Bita Moghaddam,et al.  Differences in response initiation and behavioral flexibility between adolescent and adult rats. , 2013, Behavioral neuroscience.

[61]  Michael J. Brammer,et al.  Neural and Psychological Maturation of Decision-making in Adolescence and Young Adulthood , 2013, Journal of Cognitive Neuroscience.

[62]  Catherine A. Hartley,et al.  Reinforcement learning across development: What insights can we draw from a decade of research? , 2019, Developmental Cognitive Neuroscience.

[63]  L. Steinberg Cognitive and affective development in adolescence , 2005, Trends in Cognitive Sciences.

[64]  Christopher G. Lucas,et al.  Changes in cognitive flexibility and hypothesis search across human life history from childhood to adolescence to adulthood , 2017, Proceedings of the National Academy of Sciences.

[65]  L. Wilbrecht,et al.  Adolescence and "Late Blooming" Synapses of the Prefrontal Cortex. , 2019, Cold Spring Harbor symposia on quantitative biology.