Computing Mutation Coverage in Interpolation-Based Model Checking

Coverage is a means to quantify the quality of a system specification, and is frequently applied to assess progress in system validation. Coverage is a standard measure in testing, but is very difficult to compute in the context of formal verification. We present efficient algorithms for identifying those parts of the system that are covered by a given property. Our algorithm is integrated into state-of-the-art Boolean satisfiability problem-based model checking using Craig interpolation. The key insight into our algorithm is the re-use of previously computed inductive invariants and counterexamples. This re-use permits a a rapid completion of the vast majority of tests, and enables the computation of a coverage measure with 96% accuracy with only 5× the runtime of the model checker.

[1]  Sanjai Rayadurgam,et al.  Coverage based test-case generation using model checkers , 2001, Proceedings. Eighth Annual IEEE International Conference and Workshop On the Engineering of Computer-Based Systems-ECBS 2001.

[2]  Gordon Fraser,et al.  Using Model-Checkers for Mutation-Based Test-Case Generation, Coverage Analysis and Specification Analysis , 2006, 2006 International Conference on Software Engineering Advances (ICSEA'06).

[3]  Fabio Somenzi,et al.  Dos and don'ts of CTL state coverage estimation , 2003, DAC '03.

[4]  Orna Kupferman,et al.  Coverage metrics for temporal logic model checking* , 2006, Formal Methods Syst. Des..

[5]  Guoxiang Huang,et al.  Constructing Craig Interpolation Formulas , 1995, COCOON.

[6]  Koen Claessen.  A Coverage Analysis for Safety Property Lists , 2007 .

[7]  Paul Ammann,et al.  Using model checking to generate tests from specifications , 1998, Proceedings Second International Conference on Formal Engineering Methods (Cat.No.98EX241).

[8]  Pallab Dasgupta,et al.  What lies between Design Intent Coverage and Model Checking? , 2006, Proceedings of the Design Automation & Test in Europe Conference.

[9]  Robert P. Kurshan,et al.  A Practical Approach to Coverage in Model Checking , 2001, CAV.

[10]  Pallab Dasgupta,et al.  Formal verification coverage: are the RTL-properties covering the design's architectural intent? , 2004, Proceedings Design, Automation and Test in Europe Conference and Exhibition.

[11]  Daniel Kroening,et al.  Word-Level Predicate-Abstraction and Refinement Techniques for Verifying RTL Verilog , 2008, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[12]  Ansuman Banerjee,et al.  Design-Intent Coverage - A New Paradigm for Formal Property Verification , 2006, IEEE Trans. Comput. Aided Des. Integr. Circuits Syst..

[13]  Pavel Pudlák,et al.  Lower bounds for resolution and cutting plane proofs and monotone computations , 1997, Journal of Symbolic Logic.

[14]  Ansuman Banerjee,et al.  Formal verification coverage: computing the coverage gap between temporal specifications , 2004, IEEE/ACM International Conference on Computer Aided Design, 2004. ICCAD-2004..

[15]  Joao Marques-Silva,et al.  GRASP-A new search algorithm for satisfiability , 1996, Proceedings of International Conference on Computer Aided Design.

[16]  Joseph Sifakis,et al.  Specification and verification of concurrent systems in CESAR , 1982, Symposium on Programming.

[17]  G. S. Tseitin.  On the Complexity of Derivation in Propositional Calculus , 1983 .

[18]  David A. Plaisted,et al.  A Structure-Preserving Clause Form Translation , 1986, J. Symb. Comput..

[19]  Orna Kupferman,et al.  Coverage metrics for formal verification , 2003, International Journal on Software Tools for Technology Transfer.

[20]  Armin Biere,et al.  Symbolic Model Checking without BDDs , 1999, TACAS.

[21]  Orna Kupferman,et al.  Sanity Checks in Formal Verification , 2006, CONCUR.

[22]  Sofia Cassel,et al.  Graph-Based Algorithms for Boolean Function Manipulation , 2012 .

[23]  Stephan Merz,et al.  Model Checking , 2000 .

[24]  Daniel Kroening,et al.  Test-case generation for embedded simulink via formal concept analysis , 2011, 2011 48th ACM/EDAC/IEEE Design Automation Conference (DAC).

[25]  Armin Biere,et al.  Bounded model checking , 2003, Adv. Comput..

[26]  Ansuman Banerjee,et al.  Test generation games from formal specifications , 2006, 2006 43rd ACM/IEEE Design Automation Conference.

[27]  Fred Kröger,et al.  Temporal Logic of Programs , 1987, EATCS Monographs on Theoretical Computer Science.

[28]  Hilary Putnam,et al.  A Computing Procedure for Quantification Theory , 1960, JACM.

[29]  Daniel Kroening,et al.  Interpolant Strength , 2010, VMCAI.

[30]  Daniel Kroening,et al.  Word level predicate abstraction and refinement for verifying RTL Verilog , 2005, Proceedings. 42nd Design Automation Conference, 2005..

[31]  Edmund M. Clarke,et al.  Design and Synthesis of Synchronization Skeletons Using Branching-Time Temporal Logic , 1981, Logic of Programs.

[32]  Kurt Keutzer,et al.  Coverage Metrics for Functional Validation of Hardware Designs , 2001, IEEE Des. Test Comput..

[33]  Timothy Kam,et al.  Coverage estimation for symbolic model checking , 1999, DAC '99.

[34]  Eugene Goldberg,et al.  BerkMin: A Fast and Robust Sat-Solver , 2002 .

[35]  Harry Foster,et al.  Principles of verifiable RTL design - a functional coding style supporting verification processes in Verilog , 2000 .

[36]  Kenneth L. McMillan,et al.  Interpolation and SAT-Based Model Checking , 2003, CAV.

[37]  Sanjit A. Seshia,et al.  A Theory of Mutations with Applications to Vacuity, Coverage, and Fault Tolerance , 2008, 2008 Formal Methods in Computer-Aided Design.

[38]  Orna Kupferman,et al.  Coverage of Implementations by Simulating Specifications , 2002, IFIP TCS.

[39]  Niklas Sörensson,et al.  An Extensible SAT-solver , 2003, SAT.

[40]  Jan Krajícek,et al.  Interpolation theorems, lower bounds for proof systems, and independence results for bounded arithmetic , 1997, Journal of Symbolic Logic.

[41]  Ieee Circuits,et al.  IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems information for authors , 2018, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[42]  Ansuman Banerjee,et al.  Formal methods for analyzing the completeness of an assertion suite against a high-level fault model , 2005, 18th International Conference on VLSI Design held jointly with 4th International Conference on Embedded Systems Design.

[43]  Sharad Malik,et al.  Chaff: engineering an efficient SAT solver , 2001, Proceedings of the 38th Design Automation Conference (IEEE Cat. No.01CH37232).

[44]  Orna Grumberg,et al.  "Have I written enough Properties?" - A Method of Comparison between Specification and Implementation , 1999, CHARME.

[45]  Donald W. Loveland,et al.  A machine program for theorem-proving , 2011, CACM.