Model-Based Testing and Some Steps towards Test-Based Modelling

Model-based testing is one of the promising technologies to increase the efficiency and effectiveness of software testing. In model-based testing, a model specifies the required behaviour of a system, and test cases are algorithmically generated from this model. Obtaining a valid model, however, is often difficult if the system is complex, contains legacy or third-party components, or if documentation is incomplete. Test-based modelling, also called automata learning, turns model-based testing around: it aims at automatically generating a model from test observations. This paper first gives an overview of formal, model-based testing in general, and of model-based testing for labelled transition system models in particular. Then the practice of model-based testing, the difficulty of obtaining models, and the role of learning are discussed. It is shown that model-based testing and learning are strongly related, and that learning can be fully expressed in the concepts of model-based testing. In particular, test coverage in model-based testing and precision of learned models turn out to be two sides of the same coin.

[1]  Rajeev Alur,et al.  A Temporal Logic of Nested Calls and Returns , 2004, TACAS.

[2]  Mihalis Yannakakis,et al.  Black Box Checking , 1999, FORTE.

[3]  K. Larsen,et al.  Online Testing of Real-time Systems Using Uppaal , 2004, FATES.

[4]  Carl A. Sunshine,et al.  Protocol Specification, Testing and Verification - Guest Editorial , 1982, Comput. Networks.

[5]  Ian J. Hayes,et al.  FM 2005: Formal Methods, International Symposium of Formal Methods Europe, Newcastle, UK, July 18-22, 2005, Proceedings , 2005, FM.

[6]  Boudewijn F. van Dongen,et al.  ProM: The Process Mining Toolkit , 2009, BPM.

[7]  Ed Brinksma,et al.  On the Coverage of Partial Validations , 1993, AMAST.

[8]  Erik Poll,et al.  Model-Based Testing of Electronic Passports , 2009, FMICS.

[9]  Alexandre Petrenko,et al.  Fault Model-Driven Test Derivation from Finite State Models: Annotated Bibliography , 2000, MOVEP.

[10]  Bengt Jonsson,et al.  Regular Inference for State Machines with Parameters , 2006, FASE.

[11]  Manfred Broy,et al.  Model-Based Testing of Reactive Systems, Advanced Lectures [The volume is the outcome of a research seminar that was held in Schloss Dagstuhl in January 2004] , 2005, Model-Based Testing of Reactive Systems.

[12]  Tiziana Margaria,et al.  LearnLib: a framework for extrapolating behavioral models , 2009, International Journal on Software Tools for Technology Transfer.

[13]  Axel Belinfante,et al.  Automated Testing in Practice: The Highway Tolling System , 2002, TestCom.

[14]  Jonathan Jacky,et al.  Model-Based Software Testing and Analysis with C# , 2007 .

[15]  Jan Tretmans,et al.  Test Generation with Inputs, Outputs and Repetitive Quiescence , 1996, Softw. Concepts Tools.

[16]  Marinus J. Plasmeijer,et al.  Gast: Generic Automated Software Testing , 2002, IFL.

[17]  Paul Gastin,et al.  CONCUR 2010 - Concurrency Theory, 21th International Conference, CONCUR 2010, Paris, France, August 31-September 3, 2010. Proceedings , 2010, CONCUR.

[18]  Jan Tretmans,et al.  TorX: Automated Model-Based Testing , 2003 .

[19]  Nicolae Goga,et al.  Test Selection, Trace Distance and Heuristics , 2002, TestCom.

[20]  Eugene Miya,et al.  On "Software engineering" , 1985, SOEN.

[21]  Jan Tretmans,et al.  Test Generation Based on Symbolic Specifications , 2004, FATES.

[22]  J. Tretmans.  Tangram: Model-Based Integration and Testing of Complex High-Tech Systems , 2007 .

[23]  R. G. de Vries.  Towards Formal Test Purposes , 2001 .

[24]  David Lee,et al.  Principles and methods of testing finite state machines-a survey , 1996, Proc. IEEE.

[25]  Perdita Stevens,et al.  Modelling Recursive Calls with UML State Diagrams , 2003, FASE.

[26]  Tim A. C. Willemse.  Heuristics for ioco -Based Test-Based Modelling , 2006, FMICS/PDMC.

[27]  Marie-Claude Gaudel,et al.  Testing Can Be Formal, Too , 1995, TAPSOFT.

[28]  Marius Mikucionis,et al.  Formal Methods and Testing , 2008 .

[29]  Fides Aarts.  Inference and Abstraction of Communication Protocols , 2009 .

[30]  Thierry Jéron,et al.  TGV : theory , principles and algorithms A tool for the automatic synthesis of conformance test cases for non-deterministic reactive systems , 2004 .

[31]  Mogens Nielsen,et al.  TAPSOFT '95: Theory and Practice of Software Development , 1995, Lecture Notes in Computer Science.

[32]  Leonardo Mariani,et al.  Automatic generation of software behavioral models , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[33]  Frits W. Vaandrager,et al.  Inference and Abstraction of the Biometric Passport , 2010, ISoLA.

[34]  Jan Tretmans,et al.  A Framework for Test Selection , 1991, PSTV.

[35]  Jan Tretmans,et al.  Integrating Verification, Testing, and Learning for Cryptographic Protocols , 2007, IFM.

[36]  Frits W. Vaandrager,et al.  Learning I/O Automata , 2010, CONCUR.

[37]  Rocco De Nicola,et al.  Testing Equivalences for Processes , 1984, Theor. Comput. Sci..

[38]  Bertrand Jeannet,et al.  Symbolic Test Selection Based on Approximate Analysis , 2005, TACAS.

[39]  Adam Wolisz,et al.  Testing of Communicating Systems XIV , 2002, IFIP — The International Federation for Information Processing.

[40]  Claude Jard,et al.  Modeling and Verification of Parallel Processes , 2001, Lecture Notes in Computer Science.

[41]  Hardi Hungar,et al.  Domain-Specific Optimization in Automata Learning , 2003, CAV.

[42]  Gordon Fraser,et al.  Testing - Practice and Research Techniques, 5th International Academic and Industrial Conference, TAIC PART 2010, Windsor, UK, September 3-5, 2010. Proceedings , 2010, TAIC PART.

[43]  Brian Nielsen,et al.  Formal Approaches to Software Testing, 4th International Workshop, FATES 2004, Linz, Austria, September 21, 2004, Revised Selected Papers , 2005, FATES.

[44]  Rocco De Nicola,et al.  Extensional equivalences for transition systems , 1987, Acta Informatica.

[45]  Son T. Vuong,et al.  Sensitivity analysis of the metric based test selection , 1997 .

[46]  Nancy A. Lynch,et al.  Distributed Algorithms , 1992, Lecture Notes in Computer Science.

[47]  Wolfgang Grieskamp.  Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing , 2010, TAIC PART.

[48]  Éric Piel,et al.  Observation-Based Modeling for Testing and Verifying Highly Dependable Systems – A Practitioner’s Approach , 2009 .

[49]  Boudewijn R. Haverkort,et al.  Formal Methods: Applications and Technology, 11th International Workshop, FMICS 2006 and 5th International Workshop PDMC 2006, Bonn, Germany, August 26-27, and August 31, 2006, Revised Selected Papers , 2007, FMICS/PDMC.

[50]  Benedikt Bollig,et al.  Replaying Play In and Play Out: Synthesis of Design Models from Scenarios by Learning , 2007, TACAS.

[51]  Dana Angluin,et al.  Learning Regular Sets from Queries and Counterexamples , 1987, Inf. Comput..

[52]  Marie-Claude Gaudel,et al.  Software testing based on formal specifications: a theory and a tool , 1991, Softw. Eng. J..

[53]  Olivier Charles,et al.  Relating Conformance Test Coverage to Formal Specifications , 1996, FORTE.

[54]  Margus Veanes,et al.  Testing Concurrent Object-Oriented Systems with Spec Explorer , 2005, FM.

[55]  Tiziana Margaria,et al.  Dynamic testing via automata learning , 2007, International Journal on Software Tools for Technology Transfer.

[56]  Werner Kluge,et al.  Implementation of Functional Languages , 1996, Lecture Notes in Computer Science.

[57]  Jan Tretmans,et al.  Model Based Testing with Labelled Transition Systems , 2008, Formal Methods and Testing.

[58]  Axel Belinfante,et al.  JTorX: A Tool for On-Line Model-Driven Test Derivation and Execution , 2010, TACAS.