Model-based testing for real

Model-based testing relies on abstract behavior models for test case generation. These models are abstractions, i.e., simplifications. For deterministic reactive systems, test cases are sequences of input and expected output. To bridge the different levels of abstraction, input must be concretized before being applied to the system under test. The system’s output must then be abstracted before being compared to the output of the model.The concepts are discussed along the lines of a feasibility study, an inhouse smart card case study. We describe the modeling concepts of the CASE tool AutoFocus and an approach to model-based test case generation that is based on symbolic execution with Constraint Logic Programming.Different search strategies and algorithms for test case generation are discussed. Besides validating the model itself, generated test cases were used to verify the actual hardware with respect to these traces.

[1]  Joachim Wegener,et al.  Evolutionärer Test des Zeitverhaltens von Realzeit-Systemen (Evolutionary Testing of the Temporal Behaviour of Real-Time Systems) , 2004 .

[2]  Larry Apfelbaum,et al.  Model Based Testing , 1997 .

[3]  Bernhard Schätz,et al.  Consistent Graphical Specification of Distributed Systems , 1997, FME.

[4]  Brian Nielsen.  Specification and Test of Real-Time Systems , 2000 .

[5]  Alexander Pretschner,et al.  Model based testing in evolutionary software development , 2001, Proceedings 12th International Workshop on Rapid System Prototyping. RSP 2001.

[6]  Jonathan P. Bowen,et al.  Formalization of control-flow criteria of software testing , 2001 .

[7]  Kent L. Beck,et al.  Extreme programming explained - embrace change , 1990 .

[8]  Thom W. Frühwirth,et al.  Theory and Practice of Constraint Handling Rules , 1998, J. Log. Program..

[9]  Robert V. Binder,et al.  Testing Object-Oriented Systems: Models, Patterns, and Tools , 1999 .

[10]  Sadegh Sadeghipour.  Testing cyclic software components of reactive systems on the basis of formal specifications , 1998 .

[11]  Simeon C. Ntafos,et al.  An Evaluation of Random Testing , 1984, IEEE Transactions on Software Engineering.

[12]  Lydie du Bousquet,et al.  An overview of Lutess a specification-based tool for testing synchronous software , 1999, 14th IEEE International Conference on Automated Software Engineering.

[13]  Paul Ammann,et al.  Using model checking to generate tests from specifications , 1998, Proceedings Second International Conference on Formal Engineering Methods (Cat.No.98EX241).

[14]  Alexander Pretschner,et al.  Model Based Testing for Real: The Inhouse Card Case Study , 2001 .

[15]  Elaine J. Weyuker,et al.  Axiomatizing software test data adequacy , 1986, IEEE Transactions on Software Engineering.

[16]  Hasan Ural,et al.  Formal methods for test sequence generation , 1992, Comput. Commun..

[17]  Stacy J. Prowell,et al.  Cleanroom software engineering: technology and process , 1999 .

[18]  A. Pretschner.  Classical search strategies for test case generation with Constraint Logic Programming , 2001 .

[19]  Bruno Marre,et al.  Test sequences generation from LUSTRE descriptions: GATEL , 2000, Proceedings ASE 2000. Fifteenth IEEE International Conference on Automated Software Engineering.

[20]  Elaine J. Weyuker,et al.  An Applicable Family of Data Flow Testing Criteria , 1988, IEEE Trans. Software Eng..

[21]  Lori A. Clarke,et al.  A System to Generate Test Data and Symbolically Execute Programs , 1976, IEEE Transactions on Software Engineering.

[22]  Pretschner,et al.  AutoFocus on Constraint Logic Programming , 2000 .

[23]  Alexander Pretschner.  Compositional Generation of MC/DC Integration Test Suites , 2003, Electron. Notes Theor. Comput. Sci..

[24]  Marie-Claude Gaudel,et al.  Testing Can Be Formal, Too , 1995, TAPSOFT.

[25]  Pretschner,et al.  Testing Concurrent Reactive Systems with Constraint Logic Programming , 2000 .

[26]  Thierry Jéron,et al.  Using On-The-Fly Verification Techniques for the Generation of test Suites , 1996, CAV.

[27]  Koen Claessen,et al.  QuickCheck: a lightweight tool for random testing of Haskell programs , 2000, ICFP.

[28]  Angela Greiling Keane,et al.  NO SILVER BULLET , 2003 .

[29]  Dieter Hogrefe,et al.  Autolink-a tool for automatic test generation from SDL specifications , 1998, Proceedings. 2nd IEEE Workshop on Industrial Strength Formal Specification Techniques.

[30]  Hong Zhu,et al.  Software unit test coverage and adequacy , 1997, ACM Comput. Surv..

[31]  Ioannis Parissis,et al.  Specification-based testing of synchronous software , 1996, SIGSOFT '96.

[32]  Nigel James Tracey,et al.  A search-based automated test-data generation framework for safety-critical software , 2000 .

[33]  Nicolas Halbwachs,et al.  Automatic testing of reactive systems , 1998, Proceedings 19th IEEE Real-Time Systems Symposium (Cat. No.98CB36279).

[34]  Thierry Jéron,et al.  An Approach to Symbolic Test Generation , 2000, IFM.

[35]  Simeon C. Ntafos,et al.  A Comparison of Some Structural Testing Strategies , 1988, IEEE Trans. Software Eng..

[36]  Jan Peleska,et al.  Test automation of safety-critical reactive systems , 1997 .

[37]  Thomas A. Henzinger,et al.  The Algorithmic Analysis of Hybrid Systems , 1995, Theor. Comput. Sci..

[38]  Richard G. Hamlet,et al.  Partition Testing Does Not Inspire Confidence , 1990, IEEE Trans. Software Eng..

[39]  C. V. Ramamoorthy,et al.  On the Automated Generation of Program Test Data , 1976, IEEE Transactions on Software Engineering.

[40]  Stefan Edelkamp,et al.  Directed explicit model checking with HSF-SPIN , 2001, SPIN '01.

[41]  S. L. Gerhart,et al.  Toward a theory of test data selection , 1975, IEEE Transactions on Software Engineering.

[42]  Giorgio Delzanno,et al.  Model Checking in CLP , 1999, TACAS.

[43]  W. Visser,et al.  Second Generation of a Java Model Checker , 2000 .

[44]  William E. Howden,et al.  An evaluation of the effectiveness of symbolic testing , 1978, Softw. Pract. Exp..

[45]  Winfried Lamersdorf,et al.  Extreme modeling , 2001 .

[46]  A. W. Heerink,et al.  Côte de Resyste in Progress , 2000 .

[47]  Alexandre Petrenko,et al.  Protocol testing: review of methods and relevance for software testing , 1994, ISSTA '94.

[48]  Alexander Pretschner,et al.  Model-Based Test Case Generation for Smart Cards , 2003, FMICS.

[49]  Alexander Pretschner,et al.  Model based testing in incremental system development , 2004, J. Syst. Softw..

[50]  C. R. Ramakrishnan,et al.  Logic Programming and Model Checking , 1998, PLILP/ALP.

[51]  Rachida Dssouli,et al.  Automatic Test Generation for EFSM-based Systems , 1996 .

[52]  Jan Tretmans,et al.  Test Generation with Inputs, Outputs and Repetitive Quiescence , 1996, Softw. Concepts Tools.

[53]  Alexander Pretschner,et al.  Prototype-based tests for hybrid reactive systems , 2003, 14th IEEE International Workshop on Rapid Systems Prototyping, 2003. Proceedings..

[54]  Christophe Meudec.  ATGen: automatic test data generation using constraint logic programming and symbolic execution † , 2001, Softw. Test. Verification Reliab..

[55]  Alexander Pretschner,et al.  Specification based test sequence generation with propositional logic , 2000, Softw. Test. Verification Reliab..

[56]  Jesse H. Poore,et al.  Cleanroom Software Engineering , 1996 .

[57]  Laurent Fribourg,et al.  Constraint Logic Programming Applied to Model Checking , 1999, LOPSTR.

[58]  Jonathan P. Bowen,et al.  Formalization of software testing criteria using the Z notation , 2001, 25th Annual International Computer Software and Applications Conference. COMPSAC 2001.

[59]  W. Gutjahr.  Partition Testing versus Random Testing: the Innuence of Uncertainty , 1999 .

[60]  R. Gerber,et al.  Automated symbolic analysis of reactive systems , 1998 .

[61]  David Scott Warren,et al.  Memoing for logic programs , 1992, CACM.

[62]  Tsun S. Chow,et al.  Testing Software Design Modeled by Finite-State Machines , 1978, IEEE Transactions on Software Engineering.

[63]  Alexander Pretschner,et al.  Szenarien modellbasierten Testens , 2002 .

[64]  Joachim Wegener.  Evolutionärer Test des Zeitverhaltens von Realzeit-Systemen , 2001, it Inf. Technol..

[65]  Richard Denney.  Test-case generation from Prolog-based specifications , 1991, IEEE Software.

[66]  Alex Groce,et al.  Model checking Java programs using structural heuristics , 2002, ISSTA '02.

[67]  Thom W. Frühwirth,et al.  Constraint Handling Rules , 2009, Constraint Programming.

[68]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[69]  William E. Howden,et al.  Symbolic Testing and the DISSECT Symbolic Evaluation System , 1977, IEEE Transactions on Software Engineering.

[70]  William E. Howden,et al.  Methodology for the Generation of Program Test Data , 1975, IEEE Transactions on Computers.

[71]  Walter J. Gutjahr,et al.  Partition Testing vs. Random Testing: The Influence of Uncertainty , 1999, IEEE Trans. Software Eng..