Testing with model checkers: a survey

About a decade after the initial proposal to use model checkers for the generation of test cases we take a look at the results in this field of research. Model checkers are formal verification tools, capable of providing counterexamples to violated properties. Normally, these counterexamples are meant to guide an analyst when searching for the root cause of a property violation. They are, however, also very useful as test cases. Many different approaches have been presented, many problems have been solved, yet many issues remain. This survey paper reviews the state of the art in testing with model checkers. Copyright © 2008 John Wiley & Sons, Ltd.

[1]  Gordon Fraser,et al.  Redundancy Based Test-Suite Reduction , 2007, FASE.

[2]  Patrice Godefroid,et al.  Model checking for programming languages using VeriSoft , 1997, POPL '97.

[3]  Rajeev Alur,et al.  Model-checking for real-time systems , 1990, [1990] Proceedings. Fifth Annual IEEE Symposium on Logic in Computer Science.

[4]  Sanjai Rayadurgam,et al.  Coverage based test-case generation using model checkers , 2001, Proceedings. Eighth Annual IEEE International Conference and Workshop On the Engineering of Computer-Based Systems-ECBS 2001.

[5]  Orna Kupferman,et al.  Module Checking Revisited , 1997, CAV.

[6]  Angelo Gargantini,et al.  Using model checking to generate tests from requirements specifications , 1999, ESEC/FSE-7.

[7]  Joseph Y. Halpern,et al.  Decision procedures and expressiveness in the temporal logic of branching time , 1982, STOC '82.

[8]  Wei Ding,et al.  Using a model checker to test safety properties , 2001, Proceedings Seventh IEEE International Conference on Engineering of Complex Computer Systems.

[9]  A. Jefferson Offutt,et al.  Introduction to Software Testing , 2008 .

[10]  Pierre Wolper,et al.  An Automata-Theoretic Approach to Automatic Program Verification (Preliminary Report) , 1986, LICS.

[11]  Gregg Rothermel,et al.  An empirical study of the effects of minimization on the fault detection capabilities of test suites , 1998, Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272).

[12]  Edmund M. Clarke,et al.  Model Checking , 1999, Handbook of Automated Reasoning.

[13]  Alex Groce,et al.  Adaptive Model Checking , 2002, Log. J. IGPL.

[14]  Kenneth L. McMillan,et al.  Symbolic model checking , 1992 .

[15]  Yuri Gurevich,et al.  Sequential abstract-state machines capture sequential algorithms , 2000, TOCL.

[16]  Manfred Broy,et al.  Model-Based Testing of Reactive Systems, Advanced Lectures [The volume is the outcome of a research seminar that was held in Schloss Dagstuhl in January 2004] , 2005, Model-Based Testing of Reactive Systems.

[17]  Fausto Giunchiglia,et al.  NUSMV: A New Symbolic Model Verifier , 1999, CAV.

[18]  Raymond A. Paul,et al.  Automated model checking and testing for composite Web services , 2005, Eighth IEEE International Symposium on Object-Oriented Real-Time Distributed Computing (ISORC'05).

[19]  Klaus Havelund,et al.  Model Checking Programs , 2004, Automated Software Engineering.

[20]  David L. Dill,et al.  The Murphi Verification System , 1996, CAV.

[21]  Sarfraz Khurshid,et al.  Generalized Symbolic Execution for Model Checking and Testing , 2003, TACAS.

[22]  Steven P. Miller,et al.  Applicability of modified condition/decision coverage to software testing , 1994, Softw. Eng. J..

[23]  David L. Dill,et al.  Java model checking , 2000, Proceedings ASE 2000. Fifteenth IEEE International Conference on Automated Software Engineering.

[24]  Armin Biere,et al.  Symbolic Model Checking without BDDs , 1999, TACAS.

[25]  Dexter Kozen,et al.  RESULTS ON THE PROPOSITIONAL’p-CALCULUS , 2001 .

[26]  Nikil D. Dutt,et al.  Automatic functional test program generation for pipelined processors using model checking , 2002, Seventh IEEE International High-Level Design Validation and Test Workshop, 2002..

[27]  Loe M. G. Feijs,et al.  Test Generation for Intelligent Networks Using Model Checking , 1997, TACAS.

[28]  Elaine J. Weyuker,et al.  Selecting Software Test Data Using Data Flow Information , 1985, IEEE Transactions on Software Engineering.

[29]  Magdy S. Abadir,et al.  Directed Micro-architectural Test Generation for an Industrial Processor: A Case Study , 2006, Seventh International Workshop on Microprocessor Test and Verification (MTV'06).

[30]  Bernhard K. Aichernig,et al.  Handling Model Changes: Regression Testing and Test-Suite Update with Model-Checkers , 2007, Electron. Notes Theor. Comput. Sci..

[31]  Kim G. Larsen,et al.  Time-Optimal Real-Time Test Case Generation Using Uppaal , 2003, FATES.

[32]  Wang Yi,et al.  Uppaal in a nutshell , 1997, International Journal on Software Tools for Technology Transfer.

[33]  Ajitha Rajan,et al.  Coverage metrics for requirements-based testing , 2006, ISSTA '06.

[34]  Gordon Fraser,et al.  Using Model-Checkers for Mutation-Based Test-Case Generation, Coverage Analysis and Specification Analysis , 2006, 2006 International Conference on Software Engineering Advances (ICSEA'06).

[35]  Paul E. Black.  Modeling and marshaling: making tests from model checker counterexamples , 2000, 19th DASC. 19th Digital Avionics Systems Conference. Proceedings (Cat. No.00CH37126).

[36]  Sarfraz Khurshid,et al.  Test input generation with java PathFinder , 2004, ISSTA '04.

[37]  Paul Ammann,et al.  Using model checking to generate tests from specifications , 1998, Proceedings Second International Conference on Formal Engineering Methods (Cat.No.98EX241).

[38]  Hasan Ural,et al.  Using Model Checking for Reducing the Cost of Test Generation , 2004, FATES.

[39]  Chin-Laung Lei,et al.  Modalities for Model Checking: Branching Time Logic Strikes Back , 1987, Sci. Comput. Program..

[40]  C. Eisner,et al.  Efficient Detection of Vacuity in ACTL Formulaas , 1997, CAV.

[41]  Helmut Veith,et al.  Tree-like counterexamples in model checking , 2002, Proceedings 17th Annual IEEE Symposium on Logic in Computer Science.

[42]  Edmund M. Clarke,et al.  Efficient generation of counterexamples and witnesses in symbolic model checking , 1995, DAC '95.

[43]  Edmund M. Clarke,et al.  Counterexample-Guided Abstraction Refinement , 2000, CAV.

[44]  Insup Lee,et al.  A Temporal Logic Based Theory of Test Coverage and Generation , 2002, TACAS.

[45]  Vadim Okun,et al.  Mutation operators for specifications , 2000, Proceedings ASE 2000. Fifteenth IEEE International Conference on Automated Software Engineering.

[46]  Stephen A. Edwards,et al.  The synchronous languages 12 years later , 2003, Proc. IEEE.

[47]  Corina S. Pasareanu,et al.  Test input generation for java containers using state matching , 2006, ISSTA '06.

[48]  Gordon Fraser,et al.  Using LTL rewriting to improve the performance of model-checker based test-case generation , 2007, A-MOST '07.

[49]  Paul Ammann,et al.  A specification-based coverage metric to evaluate test sets , 1999, Proceedings 4th IEEE International Symposium on High-Assurance Systems Engineering.

[50]  Susan Stepney,et al.  Challenging formal specifications by mutation: a CSP security example , 2003, Tenth Asia-Pacific Software Engineering Conference, 2003..

[51]  Grigore Rosu,et al.  Efficient monitoring of safety properties , 2004, International Journal on Software Tools for Technology Transfer.

[52]  Donglin Liang,et al.  Coverage-directed test generation with model checkers: challenges and opportunities , 2005, 29th Annual International Computer Software and Applications Conference (COMPSAC'05).

[53]  Sanjai Rayadurgam,et al.  Generating MC/DC adequate test sequences through model checking , 2003, 28th Annual NASA Goddard Software Engineering Workshop, 2003. Proceedings..

[54]  Alexandre Petrenko,et al.  Can a Model Checker Generate Tests for Non-Deterministic Systems? , 2007, MBT.

[55]  Angelo Gargantini,et al.  ASM-Based Testing: Coverage Criteria and Automatic Test Sequence , 2001, J. Univers. Comput. Sci..

[56]  Helmut Veith,et al.  Counterexamples Revisited: Principles, Algorithms, Applications , 2003, Verification: Theory and Practice.

[57]  Javier Tuya,et al.  Generating Test Cases Specifications for BPEL Compositions of Web Services Using SPIN , 2006 .

[58]  Amir Pnueli,et al.  The temporal logic of programs , 1977, 18th Annual Symposium on Foundations of Computer Science (sfcs 1977).

[59]  Gordon Fraser,et al.  Test-case prioritization with model-checkers , 2007 .

[60]  Gordon Fraser,et al.  Relating counterexamples to test cases in CTL model checking specifications , 2007, A-MOST '07.

[61]  Doron A. Peled,et al.  Model Checking and Testing Combined , 2003, ICALP.

[62]  Manfred Broy,et al.  Model-Based Testing of Reactive Systems: Advanced Lectures (Lecture Notes in Computer Science) , 2005 .

[63]  Thomas A. Henzinger,et al.  Software Verification with BLAST , 2003, SPIN.

[64]  Gerard J. Holzmann,et al.  The Model Checker SPIN , 1997, IEEE Trans. Software Eng..

[65]  Jan Tretmans,et al.  Testing Concurrent Systems: A Formal Approach , 1999, CONCUR.

[66]  F. Wotawa,et al.  Mutant Minimization for Model-Checker Based Test-Case Generation , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[67]  Joseph Sifakis,et al.  Specification and verification of concurrent systems in CESAR , 1982, Symposium on Programming.

[68]  Insup Lee,et al.  Automatic Test Generation From Statecharts Using Model Checking , 2001 .

[69]  Koushik Sen,et al.  Program monitoring with LTL in EAGLE , 2004, 18th International Parallel and Distributed Processing Symposium, 2004. Proceedings..

[70]  Mats Per Erik Heimdahl,et al.  Specification-based prototyping for embedded systems , 1999, ESEC/FSE-7.

[71]  A. Jefferson Offutt,et al.  Coverage criteria for logical expressions , 2003, 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003..

[72]  Albert Benveniste,et al.  programmi language and its , 2001 .

[73]  Edmund M. Clarke,et al.  Design and Synthesis of Synchronization Skeletons Using Branching Time Temporal Logic , 2008, 25 Years of Model Checking.

[74]  Stefania Gnesi,et al.  Witness and Counterexample Automata for ACTL , 2004, FORTE.

[75]  Alexander Pretschner,et al.  Specification based test sequence generation with propositional logic , 2000, Softw. Test. Verification Reliab..

[76]  Edmund M. Clarke,et al.  Design and Synthesis of Synchronization Skeletons Using Branching-Time Temporal Logic , 1981, Logic of Programs.

[77]  Joseph Robert Horgan,et al.  Effect of Test Set Minimization on Fault Detection Effectiveness , 1995, 1995 17th International Conference on Software Engineering.

[78]  Gregg Rothermel,et al.  Test case prioritization: an empirical study , 1999, Proceedings IEEE International Conference on Software Maintenance - 1999 (ICSM'99). 'Software Maintenance for Business Change' (Cat. No.99CB36360).

[79]  Pascal Raymond,et al.  The synchronous data flow programming language LUSTRE , 1991, Proc. IEEE.

[80]  Robin Milner,et al.  Algebraic laws for nondeterminism and concurrency , 1985, JACM.

[81]  D. Richard Kuhn,et al.  Pseudo-Exhaustive Testing for Software , 2006, 2006 30th Annual IEEE/NASA Software Engineering Workshop.

[82]  Randal E. Bryant,et al.  Graph-Based Algorithms for Boolean Function Manipulation , 1986, IEEE Transactions on Computers.

[83]  Dawson R. Engler,et al.  Proceedings of the 5th Symposium on Operating Systems Design and Implementation Cmc: a Pragmatic Approach to Model Checking Real Code , 2022 .

[84]  Thomas A. Henzinger,et al.  Generating tests from counterexamples , 2004, Proceedings. 26th International Conference on Software Engineering.

[85]  Radu Iosif,et al.  A deadlock detection tool for concurrent Java programs , 1999, Softw. Pract. Exp..

[86]  Stefan Edelkamp,et al.  Directed explicit model checking with HSF-SPIN , 2001, SPIN '01.

[87]  Wei Ding,et al.  Model Checkers in Software Testing , 2002 .

[88]  Bruno Legeard,et al.  A taxonomy of model-based testing , 2006 .

[89]  Rocco De Nicola,et al.  Action versus State based Logics for Transition Systems , 1990, Semantics of Systems of Concurrent Processes.

[90]  Leonardo de Moura,et al.  Automated Test Generation with SAL , 2005 .

[91]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[92]  Amir Pnueli,et al.  Checking that finite state concurrent programs satisfy their linear specification , 1985, POPL.

[93]  Fabio Somenzi,et al.  Vacuum Cleaning CTL Formulae , 2002, CAV.

[94]  Jr. Sheldon B. Akers,et al.  On a Theory of Boolean Functions , 1959 .

[95]  Vadim Okun,et al.  Testing with Model Checker: Insuring Fault Visibility , 2002 .

[96]  Kenneth L. McMillan,et al.  The SMV System , 1993 .

[97]  Shaoying Liu,et al.  Criteria for generating specification-based tests , 1999, Proceedings Fifth IEEE International Conference on Engineering of Complex Computer Systems (ICECCS'99) (Cat. No.PR00434).

[98]  A. Jefferson Offutt,et al.  Evaluation of three specification-based testing criteria , 2000, Proceedings Sixth IEEE International Conference on Engineering of Complex Computer Systems. ICECCS 2000.

[99]  Jing Liu,et al.  Specification-based Test Generation and Optimization Using Model Checking , 2007, First Joint IEEE/IFIP Symposium on Theoretical Aspects of Software Engineering (TASE '07).

[100]  Mats P. E. Heimdahl,et al.  Test-suite reduction for model based tests: effects on test quality and implications for testing , 2004 .

[101]  Grigore Rosu,et al.  Monitoring programs using rewriting , 2001, Proceedings 16th Annual International Conference on Automated Software Engineering (ASE 2001).

[102]  Mats Per Erik Heimdahl,et al.  Specification test coverage adequacy criteria = specification test generation inadequacy criteria , 2004, Eighth IEEE International Symposium on High Assurance Systems Engineering, 2004. Proceedings..

[103]  Gordon Fraser,et al.  Using model-checkers to generate and analyze property relevant test-cases , 2007, Software Quality Journal.

[104]  Gordon Fraser,et al.  Property relevant software testing with model-checkers , 2006, SOEN.

[105]  Timothy A. Budd,et al.  Program Testing by Specification Mutation , 1985, Comput. Lang..

[106]  Insup Lee,et al.  Data flow testing as model checking , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[107]  P. Black,et al.  Mutation of model checker specifications for test generation and evaluation , 2001 .

[108]  Nancy G. Leveson,et al.  Requirements Specification for Process-Control Systems , 1994, IEEE Trans. Software Eng..

[109]  Grigore Rosu,et al.  Rewriting-Based Techniques for Runtime Verification , 2005, Automated Software Engineering.

[110]  Gordon Fraser,et al.  Test-Case Generation and Coverage Analysis for Nondeterministic Systems Using Model-Checkers , 2007, International Conference on Software Engineering Advances (ICSEA 2007).

[111]  Daniel Kroening,et al.  A Tool for Checking ANSI-C Programs , 2004, TACAS.

[112]  A. Prasad Sistla,et al.  Automatic verification of finite state concurrent system using temporal logic specifications: a practical approach , 1983, POPL '83.

[113]  Hong Mei,et al.  An experimental comparison of four test suite reduction techniques , 2006, ICSE.

[114]  Mary Lou Soffa,et al.  A methodology for controlling the size of a test suite , 1993, TSEM.

[115]  K. Larsen,et al.  Online Testing of Real-time Systems Using Uppaal , 2004, FATES.

[116]  Angelo Gargantini,et al.  Using Spin to Generate Testsfrom ASM Specifications , 2003, Abstract State Machines.

[117]  Debra J. Richardson,et al.  Generating regression tests via model checking , 2004, Proceedings of the 28th Annual International Computer Software and Applications Conference, 2004. COMPSAC 2004..

[118]  Mary Jean Harrold,et al.  Test-Suite Reduction and Prioritization for Modified Condition/Decision Coverage , 2003, IEEE Trans. Software Eng..

[119]  Leonardo Mendonça de Moura,et al.  Generating efficient test sets with a model checker , 2004, Proceedings of the Second International Conference on Software Engineering and Formal Methods, 2004. SEFM 2004..

[120]  Gérard Berry,et al.  The Esterel Synchronous Programming Language: Design, Semantics, Implementation , 1992, Sci. Comput. Program..

[121]  Angelo Gargantini.  Using Model Checking to Generate Fault Detecting Tests , 2007, TAP.

[122]  Sanjai Rayadurgam,et al.  Test-sequence generation from formal requirement models , 2001, Proceedings Sixth IEEE International Symposium on High Assurance Systems Engineering. Special Topic: Impact of Networking.

[123]  Randal E. Bryant,et al.  Symbolic Boolean manipulation with ordered binary-decision diagrams , 1992, CSUR.

[124]  Jonathan P. Bowen,et al.  Reinforced Condition/Decision Coverage (RC/DC): A New Criterion for Software Testing , 2002, ZB.

[125]  Orna Kupferman,et al.  Vacuity Detection in Temporal Model Checking , 1999, CHARME.

[126]  Matthew B. Dwyer,et al.  Bogor: an extensible and highly-modular software model checking framework , 2003, ESEC/FSE-11.

[127]  Hasan Ural,et al.  Dependence Testing: Extending Data Flow Testing with Control Dependence , 2005, TestCom.

[128]  Insup Lee,et al.  Specification-based testing with linear temporal logic , 2004, Proceedings of the 2004 IEEE International Conference on Information Reuse and Integration, 2004. IRI 2004..

[129]  S. Ranville,et al.  Winnowing tests: Getting quality coverage from a model checker without quantity , 2001, 20th DASC. 20th Digital Avionics Systems Conference (Cat. No.01CH37219).

[130]  Klaus Havelund,et al.  Java PathFinder, A Translator from Java to Promela , 1999, SPIN.