Library learning for neurally-guided Bayesian program induction

Successful approaches to program induction require a hand-engineered domain-specific language (DSL), constraining the space of allowed programs and imparting prior knowledge of the domain. We contribute a program induction algorithm called EC2 that learns a DSL while jointly training a neural network to efficiently search for programs in the learned DSL. We use our model to synthesize functions on lists, edit text, and solve symbolic regression problems, showing how the model learns a domain-specific library of program components for expressing solutions to problems in the domain.

[1]  Michael I. Jordan,et al.  Learning Programs: A Hierarchical Bayesian Approach , 2010, ICML.

[2]  Joshua B. Tenenbaum,et al.  Human-level concept learning through probabilistic program induction , 2015, Science.

[3]  Geoffrey E. Hinton,et al.  The "wake-sleep" algorithm for unsupervised neural networks. , 1995, Science.

[4]  Noah D. Goodman,et al.  Learning Stochastic Inverses , 2013, NIPS.

[5]  Peter-Michael Osera,et al.  Type-and-example-directed program synthesis , 2015, PLDI.

[6]  Sumit Gulwani,et al.  Automating string processing in spreadsheets using input-output examples , 2011, POPL '11.

[7]  Armando Solar-Lezama,et al.  Program synthesis by sketching , 2008 .

[8]  Jürgen Schmidhuber,et al.  Optimal Ordered Problem Solver , 2002, Machine Learning.

[9]  Phil Blunsom,et al.  Inducing Tree-Substitution Grammars , 2010, J. Mach. Learn. Res..

[10]  Noah D. Goodman,et al.  Inducing Probabilistic Programs by Bayesian Program Merging , 2011, ArXiv.

[11]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[12]  Isil Dillig,et al.  Synthesizing data structure transformations from input-output examples , 2015, PLDI.

[13]  Ray J. Solomonofi,et al.  A SYSTEM FOR INCREMENTAL LEARNING BASED ON ALGORITHMIC PROBABILITY , 1989 .

[14]  Armando Solar-Lezama,et al.  Unsupervised Learning by Program Synthesis , 2015, NIPS.

[15]  Stephen Muggleton,et al.  Meta-interpretive learning of higher-order dyadic datalog: predicate invention revisited , 2013, Machine Learning.

[16]  Stephen Muggleton,et al.  Bias reformulation for one-shot function induction , 2014, ECAI.

[17]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[18]  Ute Schmid,et al.  Inductive rule learning on the knowledge level , 2011, Cognitive Systems Research.

[19]  Matthew J. Hausknecht,et al.  Neural Program Meta-Induction , 2017, NIPS.

[20]  Joshua B. Tenenbaum,et al.  Bootstrap Learning via Modular Concept Discovery , 2013, IJCAI.

[21]  R. Solomonoff.  A SYSTEM FOR INCREMENTAL LEARNING BASED ON ALGORITHMIC PROBABILITY , 1989 .

[22]  Butler W. Lampson,et al.  A Machine Learning Framework for Programming by Example , 2013, ICML.

[23]  Geoffrey E. Hinton,et al.  The Helmholtz Machine , 1995, Neural Computation.

[24]  Timothy O'Donnell,et al.  Productivity and Reuse in Language: A Theory of Linguistic Computation and Storage , 2015 .

[25]  Frank D. Wood,et al.  Inference Compilation and Universal Probabilistic Programming , 2016, AISTATS.

[26]  Armando Solar-Lezama,et al.  Learning to Infer Graphics Programs from Hand-Drawn Images , 2017, NeurIPS.

[27]  Alexander Aiken,et al.  Stochastic superoptimization , 2012, ASPLOS '13.

[28]  Sumit Gulwani,et al.  Neural-Guided Deductive Search for Real-Time Program Synthesis from Examples , 2018, ICLR.

[29]  Robert John Henderson,et al.  Cumulative learning in the lambda calculus , 2013 .

[30]  Charles A. Sutton,et al.  Mining idioms from source code , 2014, SIGSOFT FSE.

[31]  Oleksandr Polozov,et al.  Program Synthesis with Learned Code Idioms , 2018 .

[32]  Pedro M. Domingos,et al.  Programming by demonstration: a machine learning approach , 2001 .

[33]  Armando Solar-Lezama,et al.  Sampling for Bayesian Program Learning , 2016, NIPS.

[34]  Rajeev Alur,et al.  SyGuS-Comp 2016: Results and Analysis , 2016, SYNT@CAV.

[35]  Armando Solar-Lezama,et al.  Program synthesis from polymorphic refinement types , 2015, PLDI.

[36]  John R. Koza,et al.  Genetic programming - on the programming of computers by means of natural selection , 1993, Complex adaptive systems.

[37]  Sebastian Nowozin,et al.  DeepCoder: Learning to Write Programs , 2016, ICLR.

[38]  Susumu Katayama.  Towards Human-Level Inductive Functional Programming , 2015, AGI.

[39]  Li Fei-Fei,et al.  CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Pushmeet Kohli,et al.  RobustFill: Neural Program Learning under Noisy I/O , 2017, ICML.

[41]  Benjamin C. Pierce,et al.  Types and programming languages: the next generation , 2003, 18th Annual IEEE Symposium of Logic in Computer Science, 2003. Proceedings..

[42]  Sumit Gulwani,et al.  FlashMeta: a framework for inductive program synthesis , 2015, OOPSLA.