Simulation-Based Testing of Distributed Systems ; CU-CS-1004-06

Developers of distributed systems routinely construct discrete-event simulations to help them understand and evaluate the behavior of inter-component protocols. Typically written using an imperative programming language, these simulations capture basic algorithmic functionality at the same time as they focus attention on properties critical to distribution, including topology, timing, bandwidth, and overall scalability. We ask the following question: Can simulations also be used to help in the testing of distributed-system implementations? Because simulations amount to specifications of intended behavior, the code of a simulation can be viewed as an operational, albeit non-traditional, formal model. We claim that this kind of model, when used within a specification-based testing regime, provides developers with the foundations of a powerful new method for selecting effective test suites. The primary tool used in our method is a fault-based analysis of the simulation code in which a set of mutants are generated using standard codemutation techniques. The analysis can be used to rate the effectiveness of a test suite, as well as the criterion used to form it. We substantiate our claim through experiments performed on the simulations and implementations of two different distributed systems.

[1]  Elaine J. Weyuker,et al.  Automatically Generating Test Data from a Boolean Specification , 1994, IEEE Trans. Software Eng..

[2]  Keith W. Ross,et al.  Computer networking - a top-down approach featuring the internet , 2000 .

[3]  D. Richard Kuhn.  Fault classes and error detection capability of specification-based testing , 1999, TSEM.

[4]  Debra J. Richardson,et al.  Structural specification-based testing: automated support and experimental evaluation , 1999, ESEC/FSE-7.

[5]  Yanyan Wang,et al.  Automating experimentation on distributed testbeds , 2005, ASE.

[6]  Alexandre Petrenko,et al.  Protocol testing: review of methods and relevance for software testing , 1994, ISSTA '94.

[7]  Elaine J. Weyuker,et al.  An Applicable Family of Data Flow Testing Criteria , 1988, IEEE Trans. Software Eng..

[8]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[9]  Debra J. Richardson,et al.  An Analysis of Test Data Selection Criteria Using the RELAY Model of Fault Detection , 1993, IEEE Trans. Software Eng..

[10]  Shaoying Liu,et al.  Generating test data from state‐based specifications , 2003, Softw. Test. Verification Reliab..

[11]  Debra J. Richardson,et al.  Approaches to specification-based testing , 1989 .

[12]  Mark Allman,et al.  On the effective evaluation of TCP , 1999, CCRV.

[13]  Shaoying Liu,et al.  Generating test data from SOFL specifications , 1999, J. Syst. Softw..

[14]  Simone do Rocio Senger de Souza,et al.  Mutation testing applied to Estelle specifications , 2000, Proceedings of the 33rd Annual Hawaii International Conference on System Sciences.

[15]  Ian Clarke,et al.  Protecting Free Expression Online with Freenet , 2002, IEEE Internet Comput..

[16]  Phyllis G. Frankl,et al.  An Experimental Comparison of the Effectiveness of Branch Testing and Data Flow Testing , 1993, IEEE Trans. Software Eng..

[17]  Richard Lai,et al.  A survey of communication protocol testing , 2002, J. Syst. Softw..

[18]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[19]  Phyllis G. Frankl,et al.  Further empirical studies of test effectiveness , 1998, SIGSOFT '98/FSE-6.

[20]  Paulo César Masiero,et al.  Mutation testing applied to validate specifications based on statecharts , 1999, Proceedings 10th International Symposium on Software Reliability Engineering (Cat. No.PR00443).

[21]  Gregg Rothermel,et al.  An experimental determination of sufficient mutant operators , 1996, TSEM.

[22]  Alexander L. Wolf,et al.  Software testing at the architectural level , 1996, ISAW '96.

[23]  A. Jefferson Offutt,et al.  MuJava: an automated class mutation system , 2005, Softw. Test. Verification Reliab..

[24]  Henry Muccini,et al.  Using software architecture for code testing , 2004, IEEE Transactions on Software Engineering.

[25]  A. Jefferson Offutt,et al.  Deriving tests from software architectures , 2001, Proceedings 12th International Symposium on Software Reliability Engineering.

[26]  Chengjie Liu,et al.  Maintaining Strong Cache Consistency in the World Wide Web , 1998, IEEE Trans. Computers.

[27]  Michael D. Ernst,et al.  Improving test suites via operational abstraction , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[28]  Sriram Sankar,et al.  Structural specification-based testing with ADL , 1996, ISSTA '96.

[29]  Marc J. Balcer,et al.  The category-partition method for specifying and generating fuctional tests , 1988, CACM.

[30]  Paul Ammann,et al.  A SPECIFICATION-BASED COVERAGE METRIC TO EVALUATE TEST SETS , 2001 .

[31]  Lionel C. Briand,et al.  Using simulation to empirically investigate test coverage criteria based on statechart , 2004, Proceedings. 26th International Conference on Software Engineering.

[32]  Alexander L. Wolf,et al.  Acm Sigsoft Software Engineering Notes Vol 17 No 4 Foundations for the Study of Software Architecture , 2022 .

[33]  Michael D. Rice,et al.  An approach to architectural analysis and testing , 1998, ISAW '98.