A systematic review of state-based test tools

Model-based testing (MBT) is about testing a software system using a model of its behaviour. To benefit fully from MBT, automation support is required. The goal of this systematic review is determining the current state of the art of prominent MBT tool support where we focus on tools that rely on state-based models. We automatically searched different source of information including digital libraries and mailing lists dedicated to the topic. Precisely defined criteria are used to compare selected tools and comprise support for test adequacy and coverage criteria, level of automation for various testing activities and support for the construction of test scaffolding. Simple adequacy criteria are supported but not advanced ones; data(-flow) criteria are seldom supported; support for creating test scaffolding varies a great deal. The results of this review should be of interest to a wide range of stakeholders: software companies interested in selecting the most appropriate MBT tool for their needs; organizations willing to invest into creating MBT tool support; researchers interested in setting research directions.

[1]  Pieter J. Mosterman,et al.  A Taxonomy of Model-Based Testing for Embedded Systems from Multiple Industry Domains , 2011 .

[2]  T. Kanade.  Model-Based Testing of Reactive Systems , 2005 .

[3]  Paul Krause.  Software Test Automation: Effective Use of Test Execution Tools, Mark Fewster and Dorothy Graham, Addison-Wesley, 1999 (Book Review) , 2000, Softw. Test. Verification Reliab..

[4]  Alan Hartman,et al.  The AGEDIS tools for model based testing , 2004, ISSTA '04.

[5]  Avik Sinha,et al.  A measurement framework for evaluating model-based test generation tools , 2006, IBM Syst. J..

[6]  Bruno Legeard,et al.  A taxonomy of model-based testing , 2006 .

[7]  Stuart Reid,et al.  An empirical analysis of equivalence partitioning, boundary value analysis and random testing , 1997, Proceedings Fourth International Software Metrics Symposium.

[8]  Rajesh Subramanyan,et al.  A survey on model-based testing approaches: a systematic review , 2007, WEASELTech '07.

[9]  Yvan Labiche,et al.  On the Round Trip Path Testing Strategy , 2010, 2010 IEEE 21st International Symposium on Software Reliability Engineering.

[10]  Cem Kaner,et al.  Foundations of Software Testing , 2013 .

[11]  Mladen A. Vouk,et al.  Empirical studies of predicate-based software testing , 1994, Proceedings of 1994 IEEE International Symposium on Software Reliability Engineering.

[12]  Michel Diaz,et al.  Formal Description Technique Estelle: Results of the Esprit Sedos Project , 1989 .

[13]  Soumen Maity,et al.  Improved test generation algorithms for pair-wise testing , 2005, 16th IEEE International Symposium on Software Reliability Engineering (ISSRE'05).

[14]  Bernd Bruegge,et al.  Object-Oriented Software Engineering: Using UML, Patterns and Java, Second Edition , 2003 .

[15]  Nicolae Goga,et al.  Comparing TorX, Autolink, TGV and UIO Test Algorithms , 2001, SDL Forum.

[16]  William Lewis,et al.  Software Testing and Continuous Quality Improvement , 2000 .

[17]  Manfred Broy,et al.  Model-Based Testing of Reactive Systems: Advanced Lectures (Lecture Notes in Computer Science) , 2005 .

[18]  Alexander Pretschner,et al.  Model-Based Testing in Practice , 2005, FM.

[19]  William E. Lewis,et al.  Software Testing and Continuous Quality Improvement, Second Edition , 2004 .

[20]  Shaoying Liu,et al.  Generating test data from state‐based specifications , 2003, Softw. Test. Verification Reliab..

[21]  Natalia Juristo Juzgado,et al.  Maturing Software Engineering Knowledge through Classifications: A Case Study on Unit Testing Techniques , 2009, IEEE Transactions on Software Engineering.

[22]  Mauro Pezzè,et al.  Software testing and analysis - process, principles and techniques , 2007 .

[23]  Steven T. Karris.  Introduction to Stateflow with Applications , 2007 .

[24]  Robert M. Poston.  Automating specification-based software testing , 1996 .

[25]  Tadao Murata,et al.  Petri nets: Properties, analysis and applications , 1989, Proc. IEEE.

[26]  Pankaj Mudholkar,et al.  Software Testing , 2002, Computer.

[27]  Bruno Legeard,et al.  A taxonomy of model‐based testing approaches , 2012, Softw. Test. Verification Reliab..

[28]  Nicolas Halbwachs,et al.  Synchronous Programming of Reactive Systems , 1992, CAV.

[29]  Bernd Bruegge,et al.  Object-Oriented Software Engineering Using UML, Patterns, and Java , 2009 .

[30]  Christian Schallhart,et al.  Tools for Test Case Generation , 2004, Model-Based Testing of Reactive Systems.

[31]  Massimiliano Di Penta,et al.  Assessing, Comparing, and Combining State Machine-Based Testing and Structural Testing: A Series of Experiments , 2011, IEEE Transactions on Software Engineering.

[32]  Juergen Dingel,et al.  Model-Based Testing of Distributed Systems , 2008 .

[33]  K. Burr,et al.  Combinatorial Test Techniques : Table-based Automation , Test Generation and Code Coverage , 1998 .

[34]  Tore Dybå,et al.  Evidence-based software engineering , 2004, Proceedings. 26th International Conference on Software Engineering.

[35]  Boris Beizer,et al.  Software testing techniques (2. ed.) , 1990 .

[36]  Elaine J. Weyuker,et al.  On Testing Non-Testable Programs , 1982, Comput. J..

[37]  Luciano Baresi,et al.  An Introduction to Software Testing , 2006, FoVMT.

[38]  Samuel T. Chanson,et al.  A unified approach to protocol test sequence generation , 1993, IEEE INFOCOM '93 The Conference on Computer Communications, Proceedings.

[39]  Fevzi Belli,et al.  View graphs for analysis and testing of programs at different abstraction levels , 2005, Ninth IEEE International Symposium on High-Assurance Systems Engineering (HASE'05).

[40]  Lionel C. Briand,et al.  Automating regression test selection based on UML designs , 2009, Inf. Softw. Technol..

[41]  Andreas Hoffmann,et al.  Model-Based Testing , 2012, IEEE Software.

[42]  Jonas Boberg.  Early fault detection with model-based testing , 2008, ERLANG '08.

[43]  Jonathan Jacky,et al.  Model-Based Software Testing and Analysis with C# , 2007 .

[44]  Jürgen Dingel,et al.  A Survey of Using Model-Based Testing to Improve Quality Attributes in Distributed Systems , 2008, SCSS.

[45]  Mengqi Wu,et al.  Effective Software Test Automation: Developing an Automated Software Testing Tool , 2004 .

[46]  Nashat Mansour,et al.  UML-based regression testing for OO software , 2007, J. Softw. Maintenance Res. Pract..

[47]  Sigrid Eldh.  Software Testing Techniques , 2007 .

[48]  Elfriede Dustin,et al.  Implementing Automated Software Testing: How to Save Time and Lower Costs While Raising Quality , 2009 .

[49]  Jüri Vain,et al.  Model-Based Testing of Reactive Systems , 2009 .

[50]  Geoff Quentin.  Automated Software Testing: Introduction, Management and Performance, Elfriede Dustin, Jeff Rashka and John Paul, Addison-Wesley, 1999 (Book Review) , 1999, Softw. Test. Verification Reliab..

[51]  Pearl Brereton,et al.  Lessons from applying the systematic literature review process within the software engineering domain , 2007, J. Syst. Softw..

[52]  Tom Pender.  UML Bible , 2003 .

[53]  Pierre Marzin,et al.  Understanding Formal Methods , 2003, Springer London.

[54]  Paul C. Jorgensen,et al.  Software Testing: A Craftsman's Approach , 1995 .

[55]  Marc J. Balcer,et al.  The category-partition method for specifying and generating fuctional tests , 1988, CACM.

[56]  Pearl Brereton,et al.  Performing systematic literature reviews in software engineering , 2006, ICSE.

[57]  David Lee,et al.  Principles and methods of testing finite state machines-a survey , 1996, Proc. IEEE.

[58]  Tathagat Varma,et al.  Automated software testing: introduction, management and performance , 2000, SOEN.

[59]  A. Jefferson Offutt,et al.  Coverage criteria for logical expressions , 2003, 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003..

[60]  Tore Dybå,et al.  Evidence-Based Software Engineering for Practitioners , 2005, IEEE Softw..

[61]  Giuliano Antoniol,et al.  A case study using the round-trip strategy for state-based class testing , 2002, 13th International Symposium on Software Reliability Engineering, 2002. Proceedings..

[62]  Robert V. Binder,et al.  Testing Object-Oriented Systems: Models, Patterns, and Tools , 1999 .

[63]  D. M. Hutton,et al.  Software Test Automation: Effective Use of Test Execution Tools , 2000 .

[64]  Manfred Broy,et al.  Model-Based Testing of Reactive Systems, Advanced Lectures [The volume is the outcome of a research seminar that was held in Schloss Dagstuhl in January 2004] , 2005, Model-Based Testing of Reactive Systems.

[65]  Lionel C. Briand,et al.  Using simulation to empirically investigate test coverage criteria based on statechart , 2004, Proceedings. 26th International Conference on Software Engineering.

[66]  A. Jefferson Offutt,et al.  Introduction to Software Testing , 2008 .