A Comparative Study of Software Model Checkers as Unit Testing Tools: An Industrial Case Study

Conventional testing methods often fail to detect hidden flaws in complex embedded software such as device drivers or file systems. This deficiency incurs significant development and support/maintenance cost for the manufacturers. Model checking techniques have been proposed to compensate for the weaknesses of conventional testing methods through exhaustive analyses. Whereas conventional model checkers require manual effort to create an abstract target model, modern software model checkers remove this overhead by directly analyzing a target C program, and can be utilized as unit testing tools. However, since software model checkers are not fully mature yet, they have limitations according to the underlying technologies and tool implementations, potentially critical issues when applied in industrial projects. This paper reports our experience in applying Blast and CBMC to testing the components of a storage platform software for flash memory. Through this project, we analyzed the strong and weak points of two different software model checking technologies in the viewpoint of real-world industrial application-counterexample-guided abstraction refinement with predicate abstraction and SAT-based bounded analysis.

[1]  Alberto Griggio,et al.  Efficient Interpolant Generation in Satisfiability Modulo Theories , 2008, TACAS.

[2]  Gerald Lüttgen,et al.  Blasting Linux Code , 2006, FMICS/PDMC.

[3]  Daniel Jackson,et al.  Formal Modeling and Analysis of a Flash Filesystem in Alloy , 2008, ABZ.

[4]  Armando Tacchella,et al.  Theory and Applications of Satisfiability Testing , 2003, Lecture Notes in Computer Science.

[5]  Marsha Chechik,et al.  A buffer overflow benchmark for software model checkers , 2007, ASE.

[6]  Mary Sheeran,et al.  Checking Safety Properties Using Induction and a SAT-Solver , 2000, FMCAD.

[7]  Stephan Merz,et al.  Model Checking , 2000 .

[8]  Daniel Kroening,et al.  Predicate Abstraction of ANSI-C Programs Using SAT , 2004, Formal Methods Syst. Des..

[9]  Daniel Kroening,et al.  Cogent: Accurate Theorem Proving for Program Verification , 2005, CAV.

[10]  Hassen Saïdi,et al.  Construction of Abstract State Graphs with PVS , 1997, CAV.

[11]  Sriram K. Rajamani,et al.  Automatically validating temporal safety properties of interfaces , 2001, SPIN '01.

[12]  R. Joshi,et al.  Putting Flight Software Through the Paces with Testing , Model Checking , and Constraint-Solving , 2008 .

[13]  Sharad Malik,et al.  The Quest for Efficient Boolean Satisfiability Solvers , 2002, CAV.

[14]  Stefan Kowalewski,et al.  Model checking C source code for embedded systems , 2009, International Journal on Software Tools for Technology Transfer.

[15]  David Detlefs,et al.  Simplify: a theorem prover for program checking , 2005, JACM.

[16]  Daniel Kroening,et al.  A Survey of Automated Techniques for Formal Software Verification , 2008, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[17]  Daniel Kroening,et al.  A Tool for Checking ANSI-C Programs , 2004, TACAS.

[18]  Dawson R. Engler,et al.  EXE: automatically generating inputs of death , 2006, CCS '06.

[19]  Sriram K. Rajamani,et al.  SLAM and Static Driver Verifier: Technology Transfer of Formal Methods inside Microsoft , 2004, IFM.

[20]  Todd Millstein,et al.  Automatic predicate abstraction of C programs , 2001, PLDI '01.

[21]  Hong Zhu,et al.  Software unit test coverage and adequacy , 1997, ACM Comput. Surv..

[22]  Carsten Sinz,et al.  Reducing False Positives by Combining Abstract Interpretation and Bounded Model Checking , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[23]  Sivan Toledo,et al.  Algorithms and data structures for flash memories , 2005, CSUR.

[24]  Gerard J. Holzmann,et al.  A mini challenge: build a verifiable filesystem , 2007, Formal Aspects of Computing.

[25]  Daniel Kroening,et al.  SATABS: SAT-Based Predicate Abstraction for ANSI-C , 2005, TACAS.

[26]  Jun Gu,et al.  Algorithms for the satisfiability (SAT) problem: A survey , 1996, Satisfiability Problem: Theory and Applications.

[27]  Rupak Majumdar,et al.  CSIsat: Interpolation for LA+EUF , 2008, CAV.

[28]  Alex Groce,et al.  Modular verification of software components in C , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[29]  Roland J. Weiss,et al.  Applicability of the BLAST Model Checker: An Industrial Case Study , 2009, Ershov Memorial Conference.

[30]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[31]  Moonzoo Kim,et al.  Formal Verification of a Flash Memory Device Driver - An Experience Report , 2008, SPIN.

[32]  Wolfgang Mahnke,et al.  OPC Unified Architecture , 2009, Autom..

[33]  Thomas A. Henzinger,et al.  The software model checker B last : Applications to software engineering , 2007 .

[34]  Ofer Strichman,et al.  Bounded model checking , 2003, Adv. Comput..

[35]  Kenneth L. McMillan,et al.  Interpolation and SAT-Based Model Checking , 2003, CAV.

[36]  Junfeng Yang,et al.  Using model checking to find serious file system errors , 2004, TOCS.

[37]  Niklas Sörensson,et al.  An Extensible SAT-solver , 2003, SAT.

[38]  Armin Biere,et al.  Bounded Model Checking Using Satisfiability Solving , 2001, Formal Methods Syst. Des..

[39]  Kenneth L. McMillan,et al.  An interpolating theorem prover , 2005, Theor. Comput. Sci..

[40]  Moonzoo Kim,et al.  Concolic Testing of the Multi-sector Read Operation for Flash Memory File System , 2009, SBMF.

[41]  Wolfgang Küchlin,et al.  Integrated Static Analysis for Linux Device Driver Verification , 2007, IFM.

[42]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[43]  Helmut Veith,et al.  Counterexample-guided abstraction refinement for symbolic model checking , 2003, JACM.

[44]  Jim Woodcock,et al.  Mechanising a formal model of flash memory , 2009, Sci. Comput. Program..

[45]  Koushik Sen,et al.  DART: directed automated random testing , 2005, PLDI '05.

[46]  Matthias Damm,et al.  OPC Unified Architecture , 2009, Autom..

[47]  Thomas A. Henzinger,et al.  Interface Compatibility Checking for Software Modules , 2002, CAV.

[48]  Moonzoo Kim,et al.  Unit Testing of Flash Memory Device Driver through a SAT-Based Model Checker , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.