An evolutionary/heuristic-based proof searching framework for interactive theorem prover

Abstract The proof development process in interactive theorem provers (ITPs) requires the users to manually search for proofs by interacting with proof assistants. The activity of finding the correct proofs can become quite cumbersome and time consuming for users. To make the proof searching process easier in proof assistants, we provide an evolutionary/heuristic-based framework. The basic idea for the framework is to first generate random proof sequences from a population of frequently occurring proof steps that are discovered with sequential pattern mining. Generated proof sequences are then evolved till their fitness match the fitness of the target (or original) proof sequences. Three algorithms based on the proposed framework are developed using the Genetic Algorithm (GA), Simulated Annealing (SA) and Particle Swarm Optimization (PSO). Extensive experiments are performed to investigate the performance of the proposed algorithms using the HOL4 proof assistant. Results have shown that the proposed algorithms can efficiently evolve the random sequences to obtain the target sequences. In comparison, PSO performed better than SA and SA performed better than GA. In general, the experimental results suggest that combining evolutionary/heuristic algorithms with proof assistants allow efficient support for proof finding/optimization.

[1]  Tobias Nipkow,et al.  A FORMAL PROOF OF THE KEPLER CONJECTURE , 2015, Forum of Mathematics, Pi.

[2]  Sidney Amani,et al.  Cogent: Verifying High-Assurance File System Implementations , 2016, ASPLOS.

[3]  Sarah M. Loos,et al.  Graph Representations for Higher-Order Logic and Theorem Proving , 2019, AAAI.

[4]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[5]  Yves Bertot,et al.  Interactive Theorem Proving and Program Development: Coq'Art The Calculus of Inductive Constructions , 2010 .

[6]  Josef Urban,et al.  MaLARea SG1- Machine Learner for Automated Reasoning with Semantic Guidance , 2008, IJCAR.

[7]  Zhong Shao,et al.  CertiKOS: An Extensible Architecture for Building Certified Concurrent OS Kernels , 2016, OSDI.

[8]  Tobias Nipkow,et al.  Mining the Archive of Formal Proofs , 2015, CICM.

[9]  Yutaka Nagashima,et al.  Towards evolutionary theorem proving for isabelle/HOL , 2019, GECCO.

[10]  Cezary Kaliszyk,et al.  Deep Network Guided Proof Search , 2017, LPAR.

[11]  Dervis Karaboga,et al.  AN IDEA BASED ON HONEY BEE SWARM FOR NUMERICAL OPTIMIZATION , 2005 .

[12]  Jeremy Avigad,et al.  The Lean Theorem Prover (System Description) , 2015, CADE.

[13]  Ying-ping Chen,et al.  Proving theorems by using evolutionary search with human involvement , 2017, 2017 IEEE Congress on Evolutionary Computation (CEC).

[14]  Lawrence Charles Paulson,et al.  Isabelle/HOL: A Proof Assistant for Higher-Order Logic , 2002 .

[15]  Stephan Schulz,et al.  Breeding Theorem Proving Heuristics with Genetic Algorithms , 2015, GCAI.

[16]  Jian Wang,et al.  Premise Selection for Theorem Proving by Deep Graph Embedding , 2017, NIPS.

[17]  Thibault Gauthier,et al.  TacticToe: Learning to Reason with HOL4 Tactics , 2017, LPAR.

[18]  Meng Sun,et al.  Using Recurrent Neural Network to Predict Tactics for Proving Component Connector Properties in Coq , 2019, 2019 International Symposium on Theoretical Aspects of Software Engineering (TASE).

[19]  Cezary Kaliszyk,et al.  Machine Learning of Coq Proof Guidance: First Experiments , 2014, SCSS.

[20]  Jeremy Avigad,et al.  A Machine-Checked Proof of the Odd Order Theorem , 2013, ITP.

[21]  Andries Petrus Engelbrecht,et al.  Headless Chicken Particle Swarm Optimization Algorithms , 2016, ICSI.

[22]  Sarah M. Loos,et al.  HOList: An Environment for Machine Learning of Higher Order Logic Theorem Proving , 2019, ICML.

[23]  Michael Norrish,et al.  A Brief Overview of HOL4 , 2008, TPHOLs.

[24]  Francisco Herrera,et al.  A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms , 2011, Swarm Evol. Comput..

[25]  Adam Chlipala,et al.  Using Crash Hoare logic for certifying the FSCQ file system , 2015, USENIX Annual Technical Conference.

[26]  José Espírito Santo,et al.  Curry-Howard for Sequent Calculus at Last! , 2015, TLCA.

[27]  Daniel Whalen,et al.  Holophrasm: a neural Automated Theorem Prover for higher-order logic , 2016, ArXiv.

[28]  F. Wilcoxon Individual Comparisons by Ranking Methods , 1945 .

[29]  Philippe Fournier-Viger,et al.  Proof Guidance in PVS with Sequential Pattern Mining , 2019, FSEN.

[30]  Gernot Heiser,et al.  Comprehensive formal verification of an OS microkernel , 2014, TOCS.

[31]  Xi Wang,et al.  Verdi: a framework for implementing and formally verifying distributed systems , 2015, PLDI.

[32]  Chao-Hong Chen,et al.  Automatically proving mathematical theorems with evolutionary algorithms and proof assistants , 2016, 2016 IEEE Congress on Evolutionary Computation (CEC).

[33]  Melanie Mitchell,et al.  An introduction to genetic algorithms , 1996 .

[34]  Peter Rossmanith,et al.  Simulated Annealing , 2008, Taschenbuch der Algorithmen.

[35]  Kathryn A. Dowsland,et al.  Simulated Annealing , 1989, Encyclopedia of GIS.

[36]  Cezary Kaliszyk,et al.  Learning-assisted theorem proving with millions of lemmas☆ , 2015, J. Symb. Comput..

[37]  Marc Gravel,et al.  Ensuring population diversity in genetic algorithms: A technical note with application to the cell formation problem , 2007, Eur. J. Oper. Res..

[38]  J. H. Geuvers,et al.  Proof assistants: History, ideas and future , 2009 .

[39]  Sofiène Tahar,et al.  Formal Verification Methods , 2015 .

[40]  Xavier Leroy,et al.  Formal verification of a realistic compiler , 2009, CACM.

[41]  Andrew Lewis,et al.  Grey Wolf Optimizer , 2014, Adv. Eng. Softw..

[42]  Cezary Kaliszyk,et al.  HolStep: A Machine Learning Dataset for Higher-order Logic Theorem Proving , 2017, ICLR.

[43]  Francisco Herrera,et al.  A study on the use of non-parametric tests for analyzing the evolutionary algorithms’ behaviour: a case study on the CEC’2005 Special Session on Real Parameter Optimization , 2009, J. Heuristics.

[44]  Chad E. Brown,et al.  Internal Guidance for Satallax , 2016, IJCAR.

[45]  Osman Hasan,et al.  Proof searching in HOL4 with genetic algorithm , 2020, SAC.

[46]  Andrei Popescu,et al.  A Conference Management System with Verified Document Confidentiality , 2014, CAV.

[47]  Osman Hasan,et al.  Proof searching and prediction in HOL4 with evolutionary/heuristic and deep learning techniques , 2020, Applied Intelligence.

[48]  Terry Jones,et al.  Crossover, Macromutationand, and Population-Based Search , 1995, ICGA.

[49]  Ramana Kumar,et al.  CakeML: a verified implementation of ML , 2014, POPL.

[50]  Sorin Lerner,et al.  Generating correctness proofs with neural networks , 2019, MAPL@PLDI.

[51]  Hazel Duncan,et al.  The use of data-mining for the automatic formation of tactics , 2004 .

[52]  Marcel Mongeau,et al.  Simulated Annealing: From Basics to Applications , 2018, Handbook of Metaheuristics.

[53]  John R. Koza,et al.  Genetic programming - on the programming of computers by means of natural selection , 1993, Complex adaptive systems.

[54]  Josef Urban,et al.  DeepMath - Deep Sequence Models for Premise Selection , 2016, NIPS.

[55]  Jia Deng,et al.  Learning to Prove Theorems via Interacting with Proof Assistants , 2019, ICML.

[56]  Xin-She Yang,et al.  A New Metaheuristic Bat-Inspired Algorithm , 2010, NICSO.

[57]  Thibault Gauthier,et al.  Premise Selection and External Provers for HOL4 , 2015, CPP.

[58]  M. Friedman A Comparison of Alternative Tests of Significance for the Problem of $m$ Rankings , 1940 .

[59]  Ramana Kumar,et al.  A Proof Strategy Language and Proof Script Generation for Isabelle/HOL , 2016, CADE.

[60]  Lei Yu,et al.  Modelling High-Level Mathematical Reasoning in Mechanised Declarative Proofs , 2020, ArXiv.

[61]  Yun Sing Koh,et al.  A Survey of Sequential Pattern Mining , 2017 .

[62]  Bilal H. Abed-alguni,et al.  Hybridizing the Cuckoo Search Algorithm with Different Mutation Operators for Numerical Optimization Problems , 2018, J. Intell. Syst..

[63]  Cezary Kaliszyk,et al.  Reinforcement Learning of Theorem Proving , 2018, NeurIPS.

[64]  Dawn Xiaodong Song,et al.  GamePad: A Learning Environment for Theorem Proving , 2018, ICLR.

[65]  Jan Jakubuv,et al.  First Neural Conjecturing Datasets and Experiments , 2020, CICM.