Properties Incompleteness Evaluation by Functional Verification

Verification engineers cannot guarantee the correctness of the system implementation by model checking if the set of proven properties is incomplete. However, the use of model checking lacks widely accepted coverage metrics to evaluate the property completeness. The already existing metrics are based on time-consuming formal approaches that cannot be efficiently applied to medium/large systems. In this context, the paper proposes a coverage methodology based on a combination of static and dynamic verification that allows us to reduce the evaluation time with respect to pure formal approaches. The joining point between static and dynamic verification is represented by a fault model targeting functional descriptions. Functional fault simulation and dynamic automatic test pattern generation are used to quickly estimate the capability of properties in detecting functional faults. This provides a first estimation of the property completeness. Then, if necessary, model checking is used to complete the analysis, avoiding the underestimation of the property coverage that can be obtained due to the lack of exhaustiveness of dynamic verification. The proposed approach is theoretically founded and its effectiveness is compared with already existing techniques. In addition, experimental results to confirm the theoretical results are provided

[1]  Edmund M. Clarke,et al.  Characterizing Correctness Properties of Parallel Programs Using Fixpoints , 1980, ICALP.

[2]  Ian G. Harris,et al.  Fault models and test generation for hardware-software covalidation , 2003, IEEE Design & Test of Computers.

[3]  Franco Fummi,et al.  Logic-level mapping of high-level faults , 2005, Integr..

[4]  Ilan Beer,et al.  FoCs: Automatic Generation of Simulation Checkers from Formal Specifications , 2000, CAV.

[5]  Fabio Somenzi,et al.  Dos and don'ts of CTL state coverage estimation , 2003, DAC '03.

[6]  Orna Kupferman,et al.  Vacuity Detection in Temporal Model Checking , 1999, CHARME.

[7]  I.C. Teixeira,et al.  RTL-based functional test generation for high defects coverage in digital SOCs , 2000, Proceedings IEEE European Test Workshop.

[8]  Franco Fummi,et al.  AMLETO: a multi-language environment for functional test generation , 2001, Proceedings International Test Conference 2001 (Cat. No.01CH37260).

[9]  Shinji Kimura,et al.  Extended abstract: transition traversal coverage estimation for symbolic model checking , 2005, Proceedings. Second ACM and IEEE International Conference on Formal Methods and Models for Co-Design, 2005. MEMOCODE '05..

[10]  Franco Fummi,et al.  Genetic algorithms: the philosopher's stone or an effective solution for high-level TPG? , 2003, Eighth IEEE International High-Level Design Validation and Test Workshop.

[11]  C. Eisner,et al.  Efficient Detection of Vacuity in ACTL Formulaas , 1997, CAV.

[12]  Orna Kupferman,et al.  Coverage metrics for formal verification , 2004, International Journal on Software Tools for Technology Transfer.

[13]  Orna Kupferman,et al.  Coverage Metrics for Temporal Logic Model Checking , 2001, TACAS.

[14]  A. Pnueli.  The Temporal Semantics of Concurrent Programs , 1979, Theor. Comput. Sci..

[15]  C. Eisner,et al.  RuleBase: an industry-oriented formal verification tool , 1996, 33rd Design Automation Conference Proceedings, 1996.

[16]  Alexander A. Stepanov,et al.  C++ Standard Template Library , 2000 .

[17]  Franco Fummi,et al.  Redundant functional faults reduction by saboteurs synthesis [logic verification] , 2003, Eighth IEEE International High-Level Design Validation and Test Workshop.

[18]  Franco Fummi,et al.  LAERTE++: an Object Oriented High-level TPG for SystemC Designs , 2003, FDL.

[19]  Kurt Keutzer,et al.  Coverage Metrics for Functional Validation of Hardware Designs , 2001, IEEE Des. Test Comput..

[20]  Zebo Peng,et al.  High-level and hierarchical test sequence generation , 2002, Seventh IEEE International High-Level Design Validation and Test Workshop, 2002..

[21]  Shinji Kimura,et al.  Transition-based coverage estimation for symbolic model checking , 2006, Asia and South Pacific Conference on Design Automation, 2006..

[22]  Pao-Ann Hsiung,et al.  Mutation Coverage Estimation for Model Checking , 2004, ATVA.

[23]  F.Fummi Ffummi,et al.  REDUNDANT FUNCTIONAL FAULTS REDUCTION BY SABOTEURS SYNTHESIS , 2003 .

[24]  S. Kimura,et al.  Transition traversal coverage estimation for symbolic model checking , 2005, 2005 6th International Conference on ASIC.

[25]  Nikil D. Dutt,et al.  Automatic functional test program generation for pipelined processors using model checking , 2002, Seventh IEEE International High-Level Design Validation and Test Workshop, 2002..

[26]  Kenneth L. McMillan,et al.  Symbolic model checking , 1992 .

[27]  Orna Grumberg,et al.  "Have I written enough Properties?" - A Method of Comparison between Specification and Implementation , 1999, CHARME.

[28]  Amir Pnueli,et al.  Applications of Temporal Logic to the Specification and Verification of Reactive Systems: A Survey of Current Trends , 1986, Current Trends in Concurrency.

[29]  Edmund M. Clarke,et al.  Efficient generation of counterexamples and witnesses in symbolic model checking , 1995, DAC '95.

[30]  Michael J. Vilot,et al.  Standard template library , 1996 .

[31]  Fabio Somenzi,et al.  Vacuum Cleaning CTL Formulae , 2002, CAV.

[32]  Timothy Kam,et al.  Coverage estimation for symbolic model checking , 1999, DAC '99.

[33]  Fabrizio Ferrandi,et al.  Symbolic functional vector generation for VHDL specifications , 1999, Design, Automation and Test in Europe Conference and Exhibition, 1999. Proceedings (Cat. No. PR00078).

[34]  Fabio Somenzi,et al.  Logic synthesis and verification algorithms , 1996 .

[35]  Franco Fummi,et al.  A 1000X speed up for properties completeness evaluation , 2002, Seventh IEEE International High-Level Design Validation and Test Workshop, 2002..

[36]  Robert P. Kurshan,et al.  A Practical Approach to Coverage in Model Checking , 2001, CAV.