Towards Understanding How Machines Can Learn Causal Overhypotheses

Recent work in machine learning and cognitive science has suggested that understanding causal information is essential to the development of intelligence. The extensive literature in cognitive science using the “blicket detector” environment shows that children are adept at many kinds of causal inference and learning. We propose to adapt that environment for machine learning agents. One of the key challenges for current machine learning algorithms is modeling and understanding causal overhypotheses: transferable abstract hypotheses about sets of causal relationships. In contrast, even young children spontaneously learn and use causal overhypotheses. In this work, we present a new benchmark—a flexible environment which allows for the evaluation of existing techniques under variable causal overhypotheses—and demonstrate that many existing state-of-the-art methods have trouble generalizing in this environment. The code and resources for this benchmark are available at https: //github.com/CannyLab/casual_overhypotheses.

[1]  Owain Evans,et al.  Teaching Models to Express Their Uncertainty in Words , 2022, Trans. Mach. Learn. Res..

[2]  Andrew M. Dai,et al.  PaLM: Scaling Language Modeling with Pathways , 2022, J. Mach. Learn. Res..

[3]  Lisa Anne Hendricks,et al.  Training Compute-Optimal Large Language Models , 2022, ArXiv.

[4]  Jessica B. Hamrick,et al.  Learning Causal Overhypotheses through Exploration in Children and Computational Models , 2022, CLeaR.

[5]  Reza Yazdani Aminabadi,et al.  Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model , 2022, ArXiv.

[6]  Daniel McDuff,et al.  CausalCity: Complex Simulations with Agency for Causal Discovery and Reasoning , 2021, CLeaR.

[7]  Doina Precup,et al.  A Survey of Exploration Methods in Reinforcement Learning , 2021, ArXiv.

[8]  Danilo Jimenez Rezende,et al.  Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning , 2021, NeurIPS Datasets and Benchmarks.

[9]  Pieter Abbeel,et al.  Decision Transformer: Reinforcement Learning via Sequence Modeling , 2021, NeurIPS.

[10]  Weida Tong,et al.  InferBERT: A Transformer-Based Causal Inference Framework for Enhancing Pharmacovigilance , 2021, Frontiers in Artificial Intelligence.

[11]  Song-Chun Zhu,et al.  ACRE: Abstract Causal REasoning Beyond Covariation , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Yoshua Bengio,et al.  Towards Causal Representation Learning , 2021, ArXiv.

[13]  Yoshua Bengio,et al.  CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning , 2020, ICLR.

[14]  B. Schölkopf,et al.  Causal Curiosity: RL Agents Discovering Self-supervised Experiments for Causal Representation Learning , 2020, ICML.

[15]  Sergey Levine,et al.  Recurrent Independent Mechanisms , 2019, ICLR.

[16]  Zeb Kurth-Nelson,et al.  Alchemy: A structured task distribution for meta-reinforcement learning , 2021, ArXiv.

[17]  Nan Rosemary Ke,et al.  Neural Production Systems , 2021, Neural Information Processing Systems.

[18]  Mark Chen,et al.  Language Models are Few-Shot Learners , 2020, NeurIPS.

[19]  Jessica B. Hamrick,et al.  Exploring Exploration: Comparing Children with RL Agents in Unified Environments , 2020, ArXiv.

[20]  Andrew J. Davison,et al.  RLBench: The Robot Learning Benchmark & Learning Environment , 2019, IEEE Robotics and Automation Letters.

[21]  Kevin A. Smith,et al.  Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoning , 2019, Proceedings of the National Academy of Sciences.

[22]  D. Blei,et al.  Adapting Text Embeddings for Causal Inference , 2019, UAI.

[23]  Christopher Joseph Pal,et al.  A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms , 2019, ICLR.

[24]  Marlos C. Machado,et al.  Count-Based Exploration with the Successor Representation , 2018, AAAI.

[25]  S. Levine,et al.  Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning , 2019, CoRL.

[26]  Ross B. Girshick,et al.  PHYRE: A New Benchmark for Physical Reasoning , 2019, NeurIPS.

[27]  Deepak Pathak,et al.  Self-Supervised Exploration via Disagreement , 2019, ICML.

[28]  Taehoon Kim,et al.  Quantifying Generalization in Reinforcement Learning , 2018, ICML.

[29]  Amos J. Storkey,et al.  Exploration by Random Network Distillation , 2018, ICLR.

[30]  Thien Huu Nguyen,et al.  BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning , 2018, ICLR.

[31]  Ilya Sutskever,et al.  Language Models are Unsupervised Multitask Learners , 2019 .

[32]  John Schulman,et al.  Gotta Learn Fast: A New Benchmark for Generalization in RL , 2018, ArXiv.

[33]  Marlos C. Machado,et al.  Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents , 2017, J. Artif. Intell. Res..

[34]  Christopher G. Lucas,et al.  Changes in cognitive flexibility and hypothesis search across human life history from childhood to adolescence to adulthood , 2017, Proceedings of the National Academy of Sciences.

[35]  Alec Radford,et al.  Proximal Policy Optimization Algorithms , 2017, ArXiv.

[36]  Marcus Hutter,et al.  Count-Based Exploration in Feature Space for Reinforcement Learning , 2017, IJCAI.

[37]  Alexei A. Efros,et al.  Curiosity-Driven Exploration by Self-Supervised Prediction , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[38]  Marc G. Bellemare,et al.  Count-Based Exploration with Neural Density Models , 2017, ICML.

[39]  Filip De Turck,et al.  #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning , 2016, NIPS.

[40]  Tom Schaul,et al.  Unifying Count-Based Exploration and Intrinsic Motivation , 2016, NIPS.

[41]  Benjamin Van Roy,et al.  Deep Exploration via Bootstrapped DQN , 2016, NIPS.

[42]  Alex Graves,et al.  Asynchronous Methods for Deep Reinforcement Learning , 2016, ICML.

[43]  Christopher G. Lucas,et al.  When children are better (or at least more open-minded) learners than adults: Developmental differences in learning the forms of causal relationships , 2014, Cognition.

[44]  A. Gopnik.  Scientific Thinking in Young Children: Theoretical Advances, Empirical Research, and Policy Implications , 2012, Science.

[45]  Joshua B Tenenbaum,et al.  Theory-based causal induction. , 2009, Psychological review.

[46]  Peter Dayan,et al.  Q-learning , 1992, Machine Learning.

[47]  David M. Sobel,et al.  Detecting blickets: how young children use information about novel causal powers in categorization and induction. , 2000, Child development.

[48]  Jürgen Schmidhuber,et al.  Curious model-building control systems , 1991, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks.