DeepCoder: Learning to Write Programs

We develop a first line of attack for solving programming competition-style problems from input-output examples using deep learning. The approach is to train a neural network to predict properties of the program that generated the outputs from the inputs. We use the neural network’s predictions to augment search techniques from the programming languages community, including enumerative search and an SMT-based solver. Empirically, we show that our approach leads to an order of magnitude speedup over the strong non-augmented baselines and a Recurrent Neural Network approach, and that we are able to solve problems of difficulty comparable to the simplest problems on programming competition websites.

[1]  Geoffrey E. Hinton,et al.  The Helmholtz Machine , 1995, Neural Computation.

[2]  Armando Solar-Lezama,et al.  Program synthesis by sketching , 2008 .

[3]  Eyke Hüllermeier,et al.  Bayes Optimal Multilabel Classification via Probabilistic Classifier Chains , 2010, ICML.

[4]  Andrew W. Fitzgibbon,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR 2011.

[5]  Sumit Gulwani,et al.  Synthesis of loop-free programs , 2011, PLDI '11.

[6]  Eyke Hüllermeier,et al.  On label dependence and loss minimization in multi-label classification , 2012, Machine Learning.

[7]  Butler W. Lampson,et al.  A Machine Learning Framework for Programming by Example , 2013, ICML.

[8]  Daniel Tarlow,et al.  Learning to Pass Expectation Propagation Messages , 2013, NIPS.

[9]  Diederik P. Kingma,et al.  Stochastic Gradient VB and the Variational Auto-Encoder , 2013 .

[10]  Noah D. Goodman,et al.  Learning Stochastic Inverses , 2013, NIPS.

[11]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[12]  Leonidas J. Guibas,et al.  Learning Program Embeddings to Propagate Feedback on Student Code , 2015, ICML.

[13]  Jason Weston,et al.  End-To-End Memory Networks , 2015, NIPS.

[14]  Jason Weston,et al.  Memory Networks , 2014, ICLR.

[15]  Sumit Gulwani,et al.  FlashMeta: a framework for inductive program synthesis , 2015, OOPSLA.

[16]  Isil Dillig,et al.  Synthesizing data structure transformations from input-output examples , 2015, PLDI.

[17]  Sebastian Nowozin,et al.  The informed sampler: A discriminative approach to Bayesian inference in generative computer vision models , 2014, Comput. Vis. Image Underst..

[18]  Tomas Mikolov,et al.  Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets , 2015, NIPS.

[19]  Sumit Gulwani,et al.  Predicting a Correct Program in Programming by Example , 2015, CAV.

[20]  Phil Blunsom,et al.  Learning to Transduce with Unbounded Memory , 2015, NIPS.

[21]  Pushmeet Kohli,et al.  Adaptive Neural Compilation , 2016, NIPS.

[22]  Richard S. Zemel,et al.  Gated Graph Sequence Neural Networks , 2015, ICLR.

[23]  Wojciech Zaremba,et al.  Learning Simple Algorithms from Examples , 2015, ICML.

[24]  Lukasz Kaiser,et al.  Neural GPUs Learn Algorithms , 2015, ICLR.

[25]  Sergio Gomez Colmenarejo,et al.  Hybrid computing using a neural network with dynamic external memory , 2016, Nature.

[26]  Alexander Aiken,et al.  Stochastic program optimization , 2016, Commun. ACM.

[27]  Xinyun Chen.  Under Review as a Conference Paper at Iclr 2017 Delving into Transferable Adversarial Ex- Amples and Black-box Attacks , 2016 .

[28]  Marcin Andrychowicz,et al.  Neural Random Access Machines , 2015, ERCIM News.

[29]  Pushmeet Kohli,et al.  TerpreT: A Probabilistic Programming Language for Program Induction , 2016, ArXiv.

[30]  Nando de Freitas,et al.  Neural Programmer-Interpreters , 2015, ICLR.

[31]  Quoc V. Le,et al.  Neural Programmer: Inducing Latent Programs with Gradient Descent , 2015, ICLR.

[32]  Wang Ling,et al.  Latent Predictor Networks for Code Generation , 2016, ACL.

[33]  Josef Urban,et al.  DeepMath - Deep Sequence Models for Premise Selection , 2016, NIPS.

[34]  Sumit Gulwani.  Programming by examples: applications, algorithms, and ambiguity resolution , 2017, PPDP.

[35]  Tim Rocktäschel,et al.  Programming with a Differentiable Forth Interpreter , 2016, ICML.

[36]  Cezary Kaliszyk,et al.  Deep Network Guided Proof Search , 2017, LPAR.

[37]  Omer Levy,et al.  Published as a conference paper at ICLR 2018 S IMULATING A CTION D YNAMICS WITH N EURAL P ROCESS N ETWORKS , 2018 .

[38]  Sashank J. Reddi,et al.  On the Convergence of Adam and Beyond , 2018, ICLR.