Learning to solve complex tasks by growing knowledge culturally across generations

Knowledge built culturally across generations allows humans to learn far more than an individual could glean from their own experience in a lifetime. Cultural knowledge in turn rests on language: language is the richest record of what previous generations believed, valued, and practiced, and how these evolved over time. The power and mechanisms of language as a means of cultural learning, however, are not well understood, and as a result, current AI systems do not leverage language as a means for cultural knowledge transmission. Here, we take a first step towards reverse-engineering cultural learning through language. We developed a suite of complex tasks in the form of minimalist-style video games, which we deployed in an iterated learning paradigm. Human participants were limited to only two attempts (two lives) to beat each game and were allowed to write a message to a future participant who read the message before playing. Knowledge accumulated gradually across generations, allowing later generations to advance further in the games and perform more efficient actions. Multigenerational learning followed a strikingly similar trajectory to individuals learning alone with an unlimited number of lives. Successive generations of learners were able to succeed by expressing distinct types of knowledge in natural language: the dynamics of the environment, valuable goals, dangerous risks, and strategies for success. The video game paradigm we pioneer here is thus a rich test bed for developing AI systems capable of acquiring and transmitting cultural knowledge.

[1]  Jakub W. Pachocki,et al.  Dota 2 with Large Scale Deep Reinforcement Learning , 2019, ArXiv.

[2]  Rachna,et al.  Sapiens: A brief history of humankind , 2017 .

[3]  Joshua B. Tenenbaum,et al.  Building machines that learn and think like people , 2016, Behavioral and Brain Sciences.

[4]  Robert Boyd,et al.  Causal understanding is not necessary for the improvement of culturally evolving technology , 2018, Nature Human Behaviour.

[5]  K. Laland,et al.  Experimental Evidence for the Co-Evolution of Hominin Tool-Making Teaching and Language , 2014, Nature Communications.

[6]  P. Harris,et al.  Trust in Testimony: Children's Use of True and False Statements , 2004, Psychological science.

[7]  J. Stevenson.  The cultural origins of human cognition , 2001 .

[8]  Samuel J. Gershman,et al.  Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning , 2021, ArXiv.

[9]  Mark Chen,et al.  Language Models are Few-Shot Learners , 2020, NeurIPS.

[10]  T. Griffiths,et al.  Iterated learning: Intergenerational knowledge transmission reveals inductive biases , 2007, Psychonomic bulletin & review.

[11]  Michael C. Frank,et al.  Avoiding frostbite: It helps to learn from others , 2017, Behavioral and Brain Sciences.

[12]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[13]  C. Caldwell,et al.  Studying cumulative cultural evolution in the laboratory , 2008, Philosophical Transactions of the Royal Society B: Biological Sciences.

[14]  Ewan Klein,et al.  Natural Language Processing with Python , 2009 .

[15]  Alex Thornton,et al.  What is cumulative cultural evolution? , 2018, Proceedings of the Royal Society B: Biological Sciences.

[16]  Junmo Kim,et al.  A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Alexei A. Efros,et al.  Investigating Human Priors for Playing Video Games , 2018, ICML.

[18]  Joshua B. Tenenbaum,et al.  Human Learning in Atari , 2017, AAAI Spring Symposia.

[19]  A. Whiten,et al.  The multiple roles of cultural transmission experiments in understanding human cultural evolution , 2008, Philosophical Transactions of the Royal Society B: Biological Sciences.

[20]  Demis Hassabis,et al.  A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play , 2018, Science.

[21]  J. Henrich,et al.  The cultural niche: Why social learning is essential for human adaptation , 2011, Proceedings of the National Academy of Sciences.

[22]  S. Gelman.  Learning from others: children's construction of concepts. , 2009, Annual review of psychology.

[23]  F. Osiurak,et al.  Technical reasoning is important for cumulative technological culture , 2021, Nature Human Behaviour.

[24]  Tom Schaul,et al.  A video game description language for model-based or interactive learning , 2013, 2013 IEEE Conference on Computational Inteligence in Games (CIG).

[25]  Joel Z. Leibo,et al.  Open Problems in Cooperative AI , 2020, ArXiv.

[26]  Jennie Hill.  The Gates of Hell: Sir John Franklin's Tragic Quest for the North West Passage (review) , 2011 .

[27]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[28]  Wojciech M. Czarnecki,et al.  Grandmaster level in StarCraft II using multi-agent reinforcement learning , 2019, Nature.

[29]  Bowen Zhou,et al.  Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond , 2016, CoNLL.

[30]  Mario Baum,et al.  Culture And The Evolutionary Process , 2016 .

[31]  T. Griffiths,et al.  Iterated learning and the cultural ratchet , 2009 .

[32]  Honglak Lee,et al.  Deep Learning for Real-Time Atari Game Play Using Offline Monte-Carlo Tree Search Planning , 2014, NIPS.