GridTPT: a distributed platform for Theorem Prover Testing

Programming provers is a complex task; completeness or even soundness may often be broken by apparently harmless bugs. A good testing platform can contribute in detecting problems early and helping development. This paper presents GridTPT, the distributed platform for testing the veriT SMT solver. Its features are fairly standard, but it allows to easily distribute the task in a cluster. We plan to make this platform available as an open source tool for the community of developers of automated theorem provers. This presentation to PAAR’2010 will provide the opportunity to discuss the need for such a tool and the necessary features in a broader context. We would like to extract a requirement specification from this discussion, that would be useful to get dedicated implementation resources for distribution, maintenance and future development of GridTPT.

[1]  Lawrence C. Paulson,et al.  Translating Higher-Order Clauses to First-Order Clauses , 2007, Journal of Automated Reasoning.

[2]  Leonard Bolc,et al.  Intuitionistic Propositional Calculus , 1992 .

[3]  Richard J. Boulton,et al.  Theorem Proving in Higher Order Logics , 2003, Lecture Notes in Computer Science.

[4]  Arnon Avron,et al.  Decomposition Proof Systems for Gödel-Dummett Logics , 2001, Stud Logica.

[5]  Tobias Nipkow,et al.  Sledgehammer: Judgement Day , 2010, IJCAR.

[6]  Christoph Weidenbach,et al.  Computing Small Clause Normal Forms , 2001, Handbook of Automated Reasoning.

[7]  Christian G. Fermüller,et al.  Analytic Calculi for Projective Logics , 1999, TABLEAUX.

[8]  David N. Yetter,et al.  Quantales and (noncommutative) linear logic , 1990, Journal of Symbolic Logic.

[9]  Adam Pease,et al.  Towards a standard upper ontology , 2001, FOIS.

[10]  David A. McAllester.  Ontic: A Knowledge Representation System for Mathematics , 1989, CADE.

[11]  Yves Bertot,et al.  Interactive Theorem Proving and Program Development: Coq'Art The Calculus of Inductive Constructions , 2010 .

[12]  Geoff Sutcliffe,et al.  Integration of the TPTPWorld into SigmaKEE , 2008, PAAR/ESHOL.

[13]  Geoff Sutcliffe.  The 3rd IJCAR Automated Theorem Proving Competition , 2007, AI Commun..

[14]  Pascal Fontaine,et al.  veriT: An Open, Trustable and Efficient SMT-Solver , 2009, CADE.

[15]  Neil V. Murray,et al.  Prime Implicate Tries , 2009, TABLEAUX.

[16]  Georg Struth,et al.  Quantales and Temporal Logics , 2006, AMAST.

[17]  Ullrich Hustadt,et al.  Scientific Benchmarking with Temporal Logic Decision Procedures , 2002, KR.

[18]  H. Dishkant,et al.  Logic of Quantum Mechanics , 1976 .

[19]  Geoff Sutcliffe,et al.  Semantic Derivation Verification , 2005, FLAIRS Conference.

[20]  Guido Fiorino,et al.  Fast decision procedure for propositional Dummett logic based on a multiple premise tableau calculus , 2010, Inf. Sci..

[21]  J. Conway.  Regular algebra and finite machines , 1971 .

[22]  Alessandro Avellone,et al.  Optimization techniques for propositional intuitionistic logic and their implementation , 2008, Theor. Comput. Sci..

[23]  Roy Dyckhoff,et al.  A Deterministic Terminating Sequent Calculus for Gödel-Dummett logic , 1999, Log. J. IGPL.

[24]  Christoph Benzmüller,et al.  Higher-order semantics and extensionality , 2004, Journal of Symbolic Logic.

[25]  Tadeusz Strzemecki,et al.  Polynomial-time algorithms for generation of prime implicants , 1992, J. Complex..

[26]  Raymond Reiter,et al.  Foundations of Assumption-based Truth Maintenance Systems: Preliminary Report , 1987, AAAI.

[27]  J. Hurd.  First-Order Proof Tactics in Higher-Order Logic Theorem Provers In Proc , 2003 .

[28]  Richard E. Ladner,et al.  Propositional Dynamic Logic of Regular Programs , 1979, J. Comput. Syst. Sci..

[29]  Mehrnoosh Sadrzadeh,et al.  Algebra and Sequent Calculus for Epistemic Actions , 2005, LCMAS.

[30]  Geoff Sutcliffe.  System description : SystemOnTPTP , 2000 .

[31]  Peter B. Andrews.  An introduction to mathematical logic and type theory - to truth through proof , 1986, Computer science and applied mathematics.

[32]  Chad E. Brown,et al.  Analytic Tableaux for Higher-Order Logic with Choice , 2010, Journal of Automated Reasoning.

[33]  Peter Balsiger,et al.  A Benchmark Method for the Propositional Modal Logics K, KT, S4 , 2004, Journal of Automated Reasoning.

[34]  Peter B. Andrews,et al.  TPS: A hybrid automatic-interactive system for developing proofs , 2006, J. Appl. Log..

[35]  Norman W. Paton,et al.  A Foundation for the Replacement of Pipelined Physical Join Operators in Adaptive Query Processing , 2006, EDBT Workshops.

[36]  Geoff Sutcliffe.  The CADE-22 automated theorem proving system competition - CASC-22 , 2010, AI Commun..

[37]  Andrzej Szalas,et al.  Optimal Tableau Decision Procedures for PDL , 2009, ArXiv.

[38]  Journal of automated reasoning , 1986 .

[39]  Christoph Benzmüller,et al.  The LEO-II Project , 2007 .

[40]  Fabio Massacci.  Simplification: A General Constraint Propagation Technique for Propositional and Modal Tableaux , 1998, TABLEAUX.

[41]  Adam Pease,et al.  Linking Lixicons and Ontologies: Mapping WordNet to the Suggested Upper Merged Ontology , 2003, IKE.

[42]  Bob Coecke,et al.  Current research in operational quantum logic : algebras, categories, languages , 2000 .

[43]  Sandeep K. Shukla,et al.  On the Deterministic Multi-threaded Software Synthesis from Polychronous Specifications , 2008, 2008 6th ACM/IEEE International Conference on Formal Methods and Models for Co-Design.

[44]  Christian G. Fermüller,et al.  Hypersequent Calculi for Gödel Logics - a Survey , 2003, J. Log. Comput..

[45]  Cliff Hooker,et al.  The Logico-Algebraic Approach to Quantum Mechanics , 1975 .

[46]  Wolfgang Bibel,et al.  leanCoP: lean connection-based theorem proving , 2003, J. Symb. Comput..

[47]  Vasco M. Manquinho,et al.  Prime implicant computation using satisfiability algorithms , 1997, Proceedings Ninth IEEE International Conference on Tools with Artificial Intelligence.

[48]  Lawrence C. Paulson,et al.  Lightweight relevance filtering for machine-generated resolution problems , 2009, J. Appl. Log..

[49]  Jörg Hudelmaier,et al.  An O(n log n)-Space Decision Procedure for Intuitionistic Propositional Logic , 1993, J. Log. Comput..

[50]  Niklas Sörensson,et al.  An Extensible SAT-solver , 2003, SAT.

[51]  Albert Oliveras,et al.  Design and Results of the 3rd Annual Satisfiability Modulo Theories Competition (SMT-Comp 2007) , 2008, Int. J. Artif. Intell. Tools.

[52]  Stephan Schulz,et al.  E - a brainiac theorem prover , 2002, AI Commun..

[53]  Lawrence Charles Paulson,et al.  Isabelle/HOL: A Proof Assistant for Higher-Order Logic , 2002 .

[54]  Geoff Sutcliffe,et al.  THF0 - The Core of the TPTP Language for Higher-Order Logic , 2008, IJCAR.

[55]  Donald R. Morrison,et al.  PATRICIA—Practical Algorithm To Retrieve Information Coded in Alphanumeric , 1968, J. ACM.

[56]  Guido Fiorino,et al.  An O(nlog n)-SPACE Decision Procedure for the Propositional Dummett Logic , 2001, Journal of Automated Reasoning.

[57]  Lawrence J. Henschen,et al.  What Is Automated Theorem Proving? , 1985, J. Autom. Reason..

[58]  Richard Fikes,et al.  STRIPS: A New Approach to the Application of Theorem Proving to Problem Solving , 1971, IJCAI.

[59]  Geoff Sutcliffe,et al.  Progress in the Development of Automated Theorem Proving for Higher-Order Logic , 2009, CADE.

[60]  Josef Urban,et al.  MaLARea: a Metasystem for Automated Reasoning in Large Theories , 2007, ESARLT.

[61]  Mehrnoosh Sadrzadeh,et al.  Reasoning about Dynamic Epistemic Logic , 2004 .

[62]  Lawrence C. Paulson,et al.  Multimodal and intuitionistic logics in simple type theory , 2010, Log. J. IGPL.

[63]  Anthony S. Wojcik,et al.  Formal Design Verification of Digital Systems , 1983, 20th Design Automation Conference Proceedings.

[64]  Lawrence C. Paulson,et al.  The foundation of a generic theorem prover , 1989, Journal of Automated Reasoning.

[65]  Tobias Nipkow,et al.  A Tutorial Introduction to Structured Isar Proofs , 2008 .

[66]  Melvin Fitting,et al.  First-Order Logic and Automated Theorem Proving , 1990, Graduate Texts in Computer Science.

[67]  S. C. Kleene,et al.  Introduction to Metamathematics , 1952 .

[68]  Bernhard Beckert,et al.  Integrating Automated and Interactive Theorem Proving , 1998 .

[69]  George E. Collins,et al.  Partial Cylindrical Algebraic Decomposition for Quantifier Elimination , 1991, J. Symb. Comput..

[70]  Richard C. T. Lee,et al.  A New Algorithm for Generating Prime Implicants , 1970, IEEE Transactions on Computers.

[71]  Rajeev Goré,et al.  An Optimal On-the-Fly Tableau-Based Decision Procedure for PDL-Satisfiability , 2009, CADE.

[72]  Koen Claessen,et al.  Using the TPTP Language for Writing Derivations and Finite Interpretations , 2006, IJCAR.

[73]  Richard Spencer-Smith,et al.  Modal Logic , 2007 .

[74]  Reiner Hähnle,et al.  Tableaux and Related Methods , 2001, Handbook of Automated Reasoning.

[75]  Giuseppe De Giacomo,et al.  Combining Deduction and Model Checking into Tableaux and Algorithms for Converse-PDL , 2000, Inf. Comput..

[76]  Arnon Avron,et al.  A Tableau System for Gödel-Dummett Logic Based on a Hypersequent Calculus , 2000, TABLEAUX.

[77]  Peter Balsiger,et al.  Comparison of Theorem Provers for Modal Logics - Introduction and Summary , 1998, TABLEAUX.

[78]  Han-Hing Dang,et al.  Towards Algebraic Separation Logic , 2009, RelMiCS.

[79]  Teow-Hin Ngair,et al.  A New Algorithm for Incremental Prime Implicate Generation , 1993, IJCAI.

[80]  Deepak Ramachandran,et al.  First-Orderized ResearchCyc : Expressivity and Efficiency in a Common-Sense Ontology , 2005 .

[81]  Xiao-Shan Gao,et al.  Automated generation of readable proofs with geometric invariants , 1996, Journal of Automated Reasoning.

[82]  Edward Fredkin,et al.  Trie memory , 1960, Commun. ACM.

[83]  Georg Struth,et al.  On Automating the Calculus of Relations , 2008, IJCAR.

[84]  Jacques D. Fleuriot,et al.  Combining Isabelle and QEPCAD-B in the Prover's Palette , 2008, AISC/MKM/Calculemus.

[85]  George Becker,et al.  CNF and DNF Considered Harmful for Computing Prime Implicants/Implicates , 2004, Journal of Automated Reasoning.

[86]  Armin Biere,et al.  Fuzzing and delta-debugging SMT solvers , 2009, SMT '09.

[87]  Reinhold Letz,et al.  Model Elimination and Connection Tableau Procedures , 2001, Handbook of Automated Reasoning.

[88]  Hans de Nivelle,et al.  Automated Proof Construction in Type Theory Using Resolution , 2000, Journal of Automated Reasoning.

[89]  Armin Biere,et al.  PicoSAT Essentials , 2008, J. Satisf. Boolean Model. Comput..

[90]  Willard Van Orman Quine,et al.  The Problem of Simplifying Truth Functions , 1952 .

[91]  Arild Waaler,et al.  Connections in Nonclassical Logics , 2001, Handbook of Automated Reasoning.

[92]  Natarajan Shankar,et al.  PVS: Combining Specification, Proof Checking, and Model Checking , 1996, FMCAD.

[93]  Peter Höfner,et al.  Algebraic calculi for hybrid systems , 2009 .

[94]  Oliver Friedmann,et al.  A Solver for Modal Fixpoint Logics , 2010, Electron. Notes Theor. Comput. Sci..

[95]  Terry Winograd,et al.  Understanding natural language , 1974 .

[96]  Arnon Avron,et al.  Hypersequents, logical consequence and intermediate logics for concurrency , 1991, Annals of Mathematics and Artificial Intelligence.

[97]  John McCarthy,et al.  Circumscription - A Form of Non-Monotonic Reasoning , 1980, Artif. Intell..

[98]  Ulrich Furbach,et al.  An application of automated reasoning in natural language question answering , 2010, AI Commun..

[99]  Dov M. Gabbay,et al.  Modelling evolvable component systems: Part I: A logical framework , 2009, Log. J. IGPL.

[100]  Koen Claessen,et al.  New techniques that improve mace-style model nding , 2003 .

[101]  Donald W. Loveland,et al.  Mechanical Theorem-Proving by Model Elimination , 1968, JACM.

[102]  Brian Demsky.  Data structure repair using goal-directed reasoning , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[103]  Guilherme Bittencourt.  Combining Syntax and Semantics through Prime Form Representation , 2008, J. Log. Comput..

[104]  Christoph Benzmüller,et al.  Automating Quantified Multimodal Logics in Simple Type Theory -- A Case Study , 2009, ArXiv.

[105]  The Sigma Ontology Development Environment , 2003 .

[106]  M. Clavel,et al.  Principles of Maude , 1996, WRLA.

[107]  Geoff Sutcliffe.  The SZS Ontologies for Automated Reasoning Software , 2008, LPAR Workshops.

[108]  J. Davenport.  Editor , 1960 .

[109]  Tobias Nipkow,et al.  A Proof Assistant for Higher-Order Logic , 2002 .

[110]  Georg Struth,et al.  Automated verification of refinement laws , 2009, Annals of Mathematics and Artificial Intelligence.

[111]  Clark W. Barrett,et al.  The SMT-LIB Standard Version 2.0 , 2010 .

[112]  Stephan Schulz,et al.  System Description: E 0.81 , 2004, IJCAR.

[113]  Christoph Kreitz,et al.  Connection-based Theorem Proving in Classical and Non-classical Logics , 1999, J. Univers. Comput. Sci..

[114]  Rajeev Goré,et al.  The Tableau Workbench , 2009, Electron. Notes Theor. Comput. Sci..

[115]  R. Mark Greenwood,et al.  An Evolutionary Approach to Process System Development , 1999 .

[116]  Franck Cappello,et al.  Grid'5000: A Large Scale And Highly Reconfigurable Experimental Grid Testbed , 2006, Int. J. High Perform. Comput. Appl..

[117]  Dexter Kozen.  A Completeness Theorem for Kleene Algebras and the Algebra of Regular Events , 1994, Inf. Comput..

[118]  Lawrence C. Paulson,et al.  Source-Level Proof Reconstruction for Interactive Theorem Proving , 2007, TPHOLs.

[119]  D. Gabbay.  Semantical investigations in Heyting's intuitionistic logic , 1981 .

[120]  Gertrud Bauer,et al.  Calculational Reasoning Revisited (An Isabelle/Isar Experience) , 2001, TPHOLs.

[121]  Geoff Sutcliffe,et al.  The development of CASC , 2002, AI Commun..

[122]  Francis Jeffry Pelletier,et al.  Seventy-five problems for testing automatic theorem provers , 1986, Journal of Automated Reasoning.

[123]  Michael R. Genesereth,et al.  Knowledge Interchange Format , 1991, KR.

[124]  Geoff Sutcliffe,et al.  Evaluating general purpose automated theorem proving systems , 2001, Artif. Intell..

[125]  Cesare Tinelli,et al.  The SMT-LIB Standard: Version 1.2 , 2005 .

[126]  Patrick J. Hayes,et al.  A Semantics for the Knowledge Interchange Format , 2001 .

[127]  Geoff Sutcliffe,et al.  First Order Reasoning on a Large Ontology , 2007, ESARLT.

[128]  Johann Schumann,et al.  Automated Theorem Proving in Software Engineering , 2001, Springer Berlin Heidelberg.

[129]  Christopher W. Brown.  QEPCAD B: a program for computing with semi-algebraic sets using CADs , 2003, SIGS.

[130]  Alex Kean,et al.  An Incremental Method for Generating Prime Implicants/Impicates , 1990, J. Symb. Comput..

[131]  Andrei Voronkov,et al.  Limited resource strategy in resolution theorem proving , 2003, J. Symb. Comput..

[132]  Lawrence C. Paulson,et al.  Tool support for logics of programs , 1997 .

[133]  Jens Otten.  Restricting backtracking in connection calculi , 2010, AI Commun..

[134]  Edsger W. Dijkstra,et al.  Selected Writings on Computing: A personal Perspective , 1982, Texts and Monographs in Computer Science.

[135]  Nachum Dershowitz,et al.  In handbook of automated reasoning , 2001 .

[136]  Flávio Oquendo,et al.  ArchWare: Architecting Evolvable Software , 2004, EWSA.

[137]  Georg Struth,et al.  Automated Reasoning in Kleene Algebra , 2007, CADE.

[138]  Dominique Larchey-Wendling,et al.  Graph-based Decision for Gödel-Dummett Logics , 2007, Journal of Automated Reasoning.

[139]  Rajeev Goré,et al.  An On-the-fly Tableau-based Decision Procedure for PDL-satisfiability , 2009, Electron. Notes Theor. Comput. Sci..

[140]  Petr Hájek,et al.  Metamathematics of Fuzzy Logic , 1998, Trends in Logic.

[141]  Geoff Sutcliffe,et al.  Large theory reasoning with SUMO at CASC , 2010, AI Commun..

[142]  Geoff Sutcliffe,et al.  THF 0 – The Core TPTP Language for Classical Higher-Order Logic , 2007 .

[143]  Konstantin Korovin,et al.  iProver - An Instantiation-Based Theorem Prover for First-Order Logic (System Description) , 2008, IJCAR.

[144]  John Harrison,et al.  Without Loss of Generality , 2009, TPHOLs.

[145]  Pedro Quaresma,et al.  The Area Method - A Recapitulation , 2012, J. Autom. Reason..

[146]  Geoff Sutcliffe.  The TPTP Problem Library and Associated Infrastructure , 2009, Journal of Automated Reasoning.

[147]  Thomas Andreas Meyer,et al.  Implementing Iterated Belief Change Via Prime Implicates , 2007, Australian Conference on Artificial Intelligence.

[148]  Lawrence C. Paulson,et al.  Automation for interactive proof: First prototype , 2006, Inf. Comput..

[149]  Geoff Sutcliffe,et al.  An Interactive Derivation Viewer , 2007, UITP@FLoC.

[150]  Andrei Voronkov,et al.  Vampire 1.1 (System Description) , 2001, IJCAR.

[151]  Mauro Ferrari,et al.  Duplication-Free Tableau Calculi and Related Cut-Free Sequent Calculi for the Interpolable Propositional Intermediate Logics , 1999, Log. J. IGPL.

[152]  Stephen A. Cook,et al.  A Complete Axiomatization for Blocks World , 2002, J. Log. Comput..

[153]  Volker Sorge,et al.  Combined reasoning by automated cooperation , 2008, J. Appl. Log..

[154]  J. A. Robinson,et al.  A Machine-Oriented Logic Based on the Resolution Principle , 1965, JACM.

[155]  Alonzo Church,et al.  A formulation of the simple theory of types , 1940, Journal of Symbolic Logic.

[156]  K. I. Rosenthal.  Quantales and their applications , 1990 .

[157]  Peter Jackson,et al.  Computing Prime Implicants , 1990, CADE.

[158]  Geoff Sutcliffe,et al.  Automated Reasoning in Higher-Order Logic using the TPTP THF Infrastructure , 2010, J. Formaliz. Reason..

[159]  Andrei Voronkov,et al.  The design and implementation of VAMPIRE , 2002, AI Commun..

[160]  M. Baaz.  Infinite-valued Gödel logics with $0$-$1$-projections and relativizations , 1996 .

[161]  Rance Cleaveland,et al.  Implementing mathematics with the Nuprl proof development system , 1986 .

[162]  Geoff Sutcliffe,et al.  The state of CASC , 2006, AI Commun..

[163]  Christoph Kreitz,et al.  The ILTP Problem Library for Intuitionistic Logic , 2007, Journal of Automated Reasoning.

[164]  Peter J. F. Lucas,et al.  Automated Theorem Proving for Quality-checking Medical Guidelines , 2005 .

[165]  Dominique Longin,et al.  LoTREC: Logical Tableaux Research Engineering Companion , 2005, TABLEAUX.

[166]  Dominique Larchey-Wendling,et al.  Combining Proof-Search and Counter-Model Construction for Deciding Gödel-Dummett Logic , 2002, CADE.

[167]  Aaron Stump,et al.  SMT-COMP: Satisfiability Modulo Theories Competition , 2005, CAV.