Communicating Natural Programs to Humans and Machines

The Abstraction and Reasoning Corpus (ARC) is a set of procedural tasks that tests an agent’s ability to flexibly solve novel problems. While most ARC tasks are easy for humans, they are challenging for state-of-the-art AI. What makes building intelligent systems that can generalize to novel situations such as ARC difficult? We posit that the answer might be found by studying the difference of language: While humans readily generate and interpret instructions in a general language, computer systems are shackled to a narrow domain-specific language that they can precisely execute. We present LARC, the Language-complete ARC: a collection of natural language descriptions by a group of human participants who instruct each other on how to solve ARC tasks using language alone, which contains successful instructions for 88% of the ARC tasks. We analyze the collected instructions as ‘natural programs’, finding that while they resemble computer programs, they are distinct in two ways: First, they contain a wide range of primitives; Second, they frequently leverage communicative strategies beyond directly executable codes. We demonstrate that these two distinctions prevent current program synthesis techniques from leveraging LARC to its full potential, and give concrete suggestions on how to build the next-generation program synthesizers.

[1]  Yoav Artzi,et al.  Learning Compact Lexicons for CCG Semantic Parsing , 2014, EMNLP.

[2]  Armando Solar-Lezama,et al.  Write, Execute, Assess: Program Synthesis with a REPL , 2019, NeurIPS.

[3]  Illia Polosukhin,et al.  Neural Program Search: Solving Programming Tasks from Description and Examples , 2018, ICLR.

[4]  Percy Liang,et al.  Simpler Context-Dependent Logical Forms via Model Projections , 2016, ACL.

[5]  Joshua B. Tenenbaum,et al.  Learning abstract structure for drawing by efficient motor program induction , 2020, NeurIPS.

[6]  Dan Klein,et al.  Constrained Language Models Yield Few-Shot Semantic Parsers , 2021, EMNLP.

[7]  Le Song,et al.  Learning a Meta-Solver for Syntax-Guided Program Synthesis , 2018, ICLR.

[8]  Paul Portner,et al.  Formal Semantics: The Essential Readings , 2002 .

[9]  Charles Sutton,et al.  Program Synthesis with Large Language Models , 2021, ArXiv.

[10]  Joshua B. Tenenbaum,et al.  Bootstrap Learning via Modular Concept Discovery , 2013, IJCAI.

[11]  Christopher D. Manning,et al.  Learning Language Games through Interaction , 2016, ACL.

[12]  Thomas L. Griffiths,et al.  Learning Rewards from Linguistic Feedback , 2020, AAAI.

[13]  E. Spelke,et al.  Origins of knowledge. , 1992, Psychological review.

[14]  Quoc V. Le,et al.  Neural Program Synthesis with Priority Queue Training , 2018, ArXiv.

[15]  Katherine D. Kinzler,et al.  Core knowledge. , 2007, Developmental science.

[16]  Philip R. Cohen,et al.  Referring as a Collaborative Process , 2003 .

[17]  Matthew J. Hausknecht,et al.  Neural Program Meta-Induction , 2017, NIPS.

[18]  Lihong Li,et al.  Neuro-Symbolic Program Synthesis , 2016, ICLR.

[19]  S. Kay.  On the Nature of Expertise. , 1992 .

[20]  Tom M. Mitchell,et al.  Joint Concept Learning and Semantic Parsing from Natural Language Explanations , 2017, EMNLP.

[21]  Jonathan Berant,et al.  Building a Semantic Parser Overnight , 2015, ACL.

[22]  Robert D. Hawkins,et al.  Learning to communicate about shared procedural abstractions , 2021, ArXiv.

[23]  Mark Chen,et al.  Language Models are Few-Shot Learners , 2020, NeurIPS.

[24]  Benoit Baudry,et al.  A Comprehensive Study of Bloated Dependencies in the Maven Ecosystem , 2021, Empir. Softw. Eng..

[25]  Jean-Yves Audibert,et al.  Infinitely many-armed bandits , 2008, NIPS 2008.

[26]  Isil Dillig,et al.  Optimal Neural Program Synthesis from Multimodal Specifications , 2020, EMNLP.

[27]  Vladimir Solmon,et al.  The estimation of stochastic context-free grammars using the Inside-Outside algorithm , 2003 .

[28]  Sumit Gulwani,et al.  Program Synthesis Using Natural Language , 2015, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[29]  D. McDermott.  LANGUAGE OF THOUGHT , 2012 .

[30]  Jakob N. Foerster,et al.  Improving Policies via Search in Cooperative Partially Observable Games , 2019, AAAI.

[31]  Sumit Gulwani,et al.  Inductive programming meets the real world , 2015, Commun. ACM.

[32]  Sumit Gulwani,et al.  FlashMeta: a framework for inductive program synthesis , 2015, OOPSLA.

[33]  Dawn Xiaodong Song,et al.  Improving Neural Program Synthesis with Inferred Execution Traces , 2018, NeurIPS.

[34]  Isil Dillig,et al.  Benchmarking Multimodal Regex Synthesis with Complex Structures , 2020, ACL.

[35]  Colin Raffel,et al.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer , 2019, J. Mach. Learn. Res..

[36]  Miryung Kim,et al.  JShrink: in-depth investigation into debloating modern Java applications , 2020, ESEC/SIGSOFT FSE.

[37]  Steve Piantadosi,et al.  People Infer Recursive Visual Concepts from Just a Few Examples , 2019, Computational Brain & Behavior.

[38]  Pushmeet Kohli,et al.  RobustFill: Neural Program Learning under Noisy I/O , 2017, ICML.

[39]  H. H. Clark,et al.  Common ground at the understanding of demonstrative reference , 1983 .

[40]  Armando Solar-Lezama,et al.  Program Synthesis with Pragmatic Communication , 2020, NeurIPS.

[41]  Percy Liang,et al.  From Language to Programs: Bridging Reinforcement Learning and Maximum Marginal Likelihood , 2017, ACL.

[42]  Armando Solar-Lezama,et al.  DreamCoder: growing generalizable, interpretable knowledge with wake–sleep Bayesian program learning , 2020, Philosophical Transactions of the Royal Society A.

[43]  Percy Liang,et al.  Data Recombination for Neural Semantic Parsing , 2016, ACL.

[44]  Armando Solar-Lezama,et al.  Program synthesis by sketching , 2008 .

[45]  Wojciech Zaremba,et al.  Evaluating Large Language Models Trained on Code , 2021, ArXiv.

[46]  H. Harlow,et al.  The formation of learning sets. , 1949, Psychological review.

[47]  Pedro M. Domingos,et al.  Programming by Demonstration Using Version Space Algebra , 2003, Machine Learning.

[48]  Pushmeet Kohli,et al.  TerpreT: A Probabilistic Programming Language for Program Induction , 2016, ArXiv.

[49]  Percy Liang,et al.  Learning executable semantic parsers for natural language understanding , 2016, Commun. ACM.

[50]  Joshua B. Tenenbaum,et al.  Building machines that learn and think like people , 2016, Behavioral and Brain Sciences.

[51]  A. Tate.  A measure of intelligence , 2012 .

[52]  F. Bartlett,et al.  Remembering: A Study in Experimental and Social Psychology , 1932 .

[53]  Armando Solar-Lezama,et al.  Learning to Infer Program Sketches , 2019, ICML.

[54]  Dan Klein,et al.  Learning with Latent Language , 2017, NAACL.

[55]  Todd M. Gureckis,et al.  Fast and flexible: Human program induction in abstract reasoning tasks , 2021, ArXiv.

[56]  T. Lombrozo.  The structure and function of explanations , 2006, Trends in Cognitive Sciences.

[57]  Sebastian Nowozin,et al.  DeepCoder: Learning to Write Programs , 2016, ICLR.

[58]  Percy Liang,et al.  SPoC: Search-based Pseudocode to Code , 2019, NeurIPS.

[59]  Andrea Madotto,et al.  Language Models as Few-Shot Learner for Task-Oriented Dialogue Systems , 2020, ArXiv.

[60]  Xiangping Chen,et al.  Does your code need comment? , 2020, Softw. Pract. Exp..

[61]  Jacob Andreas,et al.  Unnatural Language Processing: Bridging the Gap Between Synthetic and Natural Language Data , 2020, ArXiv.

[62]  Sanjit A. Seshia,et al.  Combinatorial sketching for finite programs , 2006, ASPLOS XII.

[63]  Demis Hassabis,et al.  Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm , 2017, ArXiv.

[64]  Luke S. Zettlemoyer,et al.  Weakly Supervised Learning of Semantic Parsers for Mapping Instructions to Actions , 2013, TACL.