Using Model Checking to Generate Fault Detecting Tests

We present a technique which generates from Abstract State Machines specifications a set of test sequences capable to uncover specific fault classes. The notion of test goal is introduced as a state predicate denoting the detection condition for a particular fault. Tests are generated by forcing a model checker to produce counter examples which cover the test goals. We introduce a technique for the evaluation of the fault detection capability of a test set. We report some experimental results which validate the method, compare the fault adequacy criteria with some classical structural coverage criteria and show an empirical cross coverage among faults.

[1]  Tsong Yueh Chen,et al.  Test case selection strategies based on Boolean specifications ‡ , 2001, Softw. Test. Verification Reliab..

[2]  Mats P. E. Heimdahl,et al.  Test-suite reduction for model based tests: effects on test quality and implications for testing , 2004 .

[3]  Doo-Hwan Bae,et al.  Test cases generation from UML state diagrams , 1999, IEE Proc. Softw..

[4]  Thomas A. Henzinger,et al.  Generating tests from counterexamples , 2004, Proceedings. 26th International Conference on Software Engineering.

[5]  Hong Zhu,et al.  Software unit test coverage and adequacy , 1997, ACM Comput. Surv..

[6]  Rajeev Alur,et al.  A Temporal Logic of Nested Calls and Returns , 2004, TACAS.

[7]  Wolfram Schulte,et al.  The ABCs of specification: asml, behavior, and components , 2001, Informatica.

[8]  Paul Ammann,et al.  Using model checking to generate tests from specifications , 1998, Proceedings Second International Conference on Formal Engineering Methods (Cat.No.98EX241).

[9]  D. Richard Kuhn.  Fault classes and error detection capability of specification-based testing , 1999, TSEM.

[10]  Debra J. Richardson,et al.  Structural specification-based testing: automated support and experimental evaluation , 1999, ESEC/FSE-7.

[11]  David Lee,et al.  Principles and methods of testing finite state machines-a survey , 1996, Proc. IEEE.

[12]  Jr. Sheldon B. Akers,et al.  On a Theory of Boolean Functions , 1959 .

[13]  Alexander Pretschner,et al.  Model-Based Testing in Practice , 2005, FM.

[14]  David Lorge Parnas,et al.  Documentation for safety critical software , 1993, Proceedings of 1993 15th International Conference on Software Engineering.

[15]  Jonathan P. Bowen,et al.  A formal analysis of MCDC and RCDC test criteria: Research Articles , 2005 .

[16]  Wei Ding,et al.  Model Checkers in Software Testing , 2002 .

[17]  Vadim Okun,et al.  Comparison of fault classes in specification-based testing , 2004, Inf. Softw. Technol..

[18]  Joanne M. Atlee,et al.  A logic-model semantics for SCR software requirements , 1996, ISSTA '96.

[19]  Ian J. Hayes,et al.  FM 2005: Formal Methods, International Symposium of Formal Methods Europe, Newcastle, UK, July 18-22, 2005, Proceedings , 2005, FM.

[20]  R.A. DeMillo,et al.  An extended overview of the Mothra software testing environment , 1988, [1988] Proceedings. Second Workshop on Software Testing, Verification, and Analysis.

[21]  Jonathan P. Bowen,et al.  A formal analysis of MCDC and RCDC test criteria , 2005, Softw. Test. Verification Reliab..

[22]  Angelo Gargantini,et al.  Using Spin to Generate Testsfrom ASM Specifications , 2003, Abstract State Machines.

[23]  Nancy G. Leveson,et al.  An empirical evaluation of the MC/DC coverage criterion on the HETE-2 satellite software , 2000, 19th DASC. 19th Digital Avionics Systems Conference. Proceedings (Cat. No.00CH37126).

[24]  Elaine J. Weyuker,et al.  Automatically Generating Test Data from a Boolean Specification , 1994, IEEE Trans. Software Eng..

[25]  Leonardo Mendonça de Moura,et al.  Generating efficient test sets with a model checker , 2004, Proceedings of the Second International Conference on Software Engineering and Formal Methods, 2004. SEFM 2004..

[26]  Man F. Lau,et al.  An extended fault class hierarchy for specification-based testing , 2005, TSEM.

[27]  Insup Lee,et al.  Data flow testing as model checking , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[28]  Angelo Gargantini,et al.  Abstract State Machines 2003 , 2003, Lecture Notes in Computer Science.

[29]  P. Black,et al.  Mutation of model checker specifications for test generation and evaluation , 2001 .

[30]  Brian Nielsen,et al.  Formal Approaches to Software Testing, 4th International Workshop, FATES 2004, Linz, Austria, September 21, 2004, Revised Selected Papers , 2005, FATES.

[31]  Egon Börger,et al.  Abstract State Machines. A Method for High-Level System Design and Analysis , 2003 .

[32]  Angelo Gargantini,et al.  Using model checking to generate tests from requirements specifications , 1999, ESEC/FSE-7.

[33]  Sanjai Rayadurgam,et al.  Generating MC/DC adequate test sequences through model checking , 2003, 28th Annual NASA Goddard Software Engineering Workshop, 2003. Proceedings..

[34]  Insup Lee,et al.  Automatic Test Generation From Statecharts Using Model Checking , 2001 .

[35]  Robert M. Hierons.  Comparing test sets and criteria in the presence of test hypotheses and fault domains , 2002, TSEM.

[36]  Loe M. G. Feijs,et al.  Test Generation for Intelligent Networks Using Model Checking , 1997, TACAS.

[37]  Siddhartha R. Dalal,et al.  Model-based testing in practice , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[38]  Paul Ammann,et al.  A SPECIFICATION-BASED COVERAGE METRIC TO EVALUATE TEST SETS , 2001 .

[39]  Jonathan P. Bowen,et al.  Ordering Mutants to Minimise Test Effort in Mutation Testing , 2004, FATES.

[40]  Elaine J. Weyuker,et al.  A Formal Analysis of the Fault-Detecting Ability of Testing Methods , 1993, IEEE Trans. Software Eng..

[41]  Angelo Gargantini,et al.  ASM-Based Testing: Coverage Criteria and Automatic Test Sequence , 2001, J. Univers. Comput. Sci..

[42]  Gerard J. Holzmann,et al.  The Model Checker SPIN , 1997, IEEE Trans. Software Eng..

[43]  Tatsuhiro Tsuchiya,et al.  On fault classes and error detection capability of specification-based testing , 2002, TSEM.