Counterexample-Driven Genetic Programming: Heuristic Program Synthesis from Formal Specifications

Conventional genetic programming (GP) can guarantee only that synthesized programs pass tests given by the provided input-output examples. The alternative to such a test-based approach is synthesizing programs by formal specification, typically realized with exact, nonheuristic algorithms. In this article, we build on our earlier study on Counterexample-Based Genetic Programming (CDGP), an evolutionary heuristic that synthesizes programs from formal specifications. The candidate programs in CDGP undergo formal verification with a Satisfiability Modulo Theory (SMT) solver, which results in counterexamples that are subsequently turned into tests and used to calculate fitness. The original CDGP is extended here with a fitness threshold parameter that decides which programs should be verified, a more rigorous mechanism for turning counterexamples into tests, and other conceptual and technical improvements. We apply it to 24 benchmarks representing two domains: the linear integer arithmetic (LIA) and the string manipulation (SLIA) problems, showing that CDGP can reliably synthesize provably correct programs in both domains. We also confront it with two state-of-the art exact program synthesis methods and demonstrate that CDGP effectively trades longer synthesis time for smaller program size.

[1]  Lishan Kang,et al.  Hoare logic-based genetic programming , 2011, Science China Information Sciences.

[2]  Alexander E. I. Brownlee,et al.  Object-Oriented Genetic Improvement for Improved Energy Consumption in Google Guava , 2015, SSBSE.

[3]  Krzysztof Krawiec,et al.  Counterexample-driven genetic programming , 2017, GECCO.

[4]  Martin Hofmann,et al.  IGOR2 - an analytical inductive functional programming system: tool demo , 2010, PEPM '10.

[5]  Rajeev Alur,et al.  Results and Analysis of SyGuS-Comp'15 , 2015, SYNT.

[6]  Martin Hofmann.  IGORII - an Analytical Inductive Functional Programming System , 2010 .

[7]  Colin G. Johnson,et al.  Genetic Programming with Fitness Based on Model Checking , 2007, EuroGP.

[8]  Mark Harman,et al.  Babel Pidgin: SBSE Can Grow and Graft Entirely New Functionality into a Real World System , 2014, SSBSE.

[9]  Clark W. Barrett,et al.  The SMT-LIB Standard Version 2.0 , 2010 .

[10]  Jerry Swan,et al.  Asymptotic Genetic Improvement Programming via Type Functors and Catamorphisms , 2014 .

[11]  Arjun Radhakrishna,et al.  Scaling Enumerative Program Synthesis via Divide and Conquer , 2017, TACAS.

[12]  Yvan Labiche,et al.  Search-Based Software Engineering : 7th International Symposium, SSBSE 2015, Bergamo, Italy, September 5-7, 2015, Proceedings , 2015 .

[13]  Jerry Swan,et al.  Automatic Improvement of Apache Spark Queries using Semantics-preserving Program Reduction , 2016, GECCO.

[14]  Stephen Muggleton.  Inductive Logic Programming: Derivations, Successes and Shortcomings , 1993, ECML.

[15]  Sanjit A. Seshia,et al.  Combinatorial sketching for finite programs , 2006, ASPLOS XII.

[16]  Sumit Gulwani,et al.  Dimensions in program synthesis , 2010, Formal Methods in Computer Aided Design.

[17]  Doron A. Peled,et al.  Genetic Programming and Model Checking: Synthesizing New Mutual Exclusion Algorithms , 2008, ATVA.

[18]  Mukund Raghothaman,et al.  Language to Specify Syntax-Guided Synthesis Problems , 2014, ArXiv.

[19]  Sumit Gulwani,et al.  Oracle-guided component-based program synthesis , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[20]  Patricia S. O Sullivan,et al.  100 Statistical Tests , 1995 .

[21]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.

[22]  Jerry Swan,et al.  Genetic Programming \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varvec{+}$$\end{document}+ Proof Search \document , 2017, Journal of Automated Reasoning.

[23]  Krzysztof Krawiec,et al.  Behavioral Program Synthesis with Genetic Programming , 2015, Studies in Computational Intelligence.

[24]  Xin Yao,et al.  Co-evolutionary automatic programming for software development , 2014, Inf. Sci..

[25]  Lee Spector,et al.  Solving Uncompromising Problems With Lexicase Selection , 2015, IEEE Transactions on Evolutionary Computation.

[26]  Susumu Katayama.  An analytical inductive functional programming system that avoids unintended programs , 2012, PEPM '12.

[27]  Sumit Gulwani,et al.  From program verification to program synthesis , 2010, POPL '10.

[28]  Sean Luke,et al.  Lexicographic Parsimony Pressure , 2002, GECCO.

[29]  Jonathan P. Bowen,et al.  Formal Methods: State of the Art and New Directions , 2009 .

[30]  Rajeev Alur,et al.  Syntax-guided synthesis , 2013, 2013 Formal Methods in Computer-Aided Design.

[31]  Doron A. Peled,et al.  Synthesis of Parametric Programs using Genetic Programming and Model Checking , 2014, INFINITY.

[32]  D. Wolfe,et al.  Nonparametric Statistical Methods. , 1974 .

[33]  Viktor Kuncak,et al.  Refutation-based synthesis in SMT , 2019, Formal Methods Syst. Des..

[34]  Xin Yao,et al.  Coevolving programs and unit tests from their specification , 2007, ASE.

[35]  Ute Schmid,et al.  Analysis and Evaluation of Inductive Programming Systems in a Higher-Order Framework , 2008, KI.

[36]  Doron A. Peled,et al.  Synthesizing, Correcting and Improving Code, Using Model Checking-Based Genetic Programming , 2013, Haifa Verification Conference.

[37]  Jordan B. Pollack,et al.  Pareto Optimality in Coevolutionary Learning , 2001, ECAL.

[38]  José B. Almeida,et al.  An Overview of Formal Methods Tools and Techniques , 2011 .

[39]  Krzysztof Krawiec,et al.  Evolutionary Program Sketching , 2017, EuroGP.

[40]  Kent L. Beck,et al.  Test-driven Development - by example , 2002, The Addison-Wesley signature series.

[41]  C. A. R. HOARE,et al.  An axiomatic basis for computer programming , 1969, CACM.

[42]  Sumit Gulwani,et al.  Component-based Synthesis Applied to Bitvector Programs , 2010 .

[43]  Gopal Kanji,et al.  100 Statistical Tests , 1994 .

[44]  Rose F. Gamble,et al.  Entailment for specification refinement , 1996 .