Simulation-Based Testing of Distributed Systems

Developers of distributed systems routinely construct discrete-event simulations to help them understand and evaluate the behavior of inter-component protocols. Typically written using an imperative programming language, these simulations capture basic algorithmic functionality at the same time as they focus attention on properties critical to distribution, including topology, timing, bandwidth, and overall scalability. We ask the following question: Can simulations also be used to help in the testing of distributed-system implementations? Because simulations amount to specifications of intended behavior, the code of a simulation can be viewed as an operational, albeit non-traditional, formal model. We claim that this kind of model, when used within a specification-based testing regime, provides developers with the foundations of a powerful new method for selecting eective test suites. The primary tool used in our method is a fault-based analysis of the simulation code in which a set of mutants are generated using standard codemutation techniques. The analysis can be used to rate the eectiveness of a test suite, as well as the criterion used to form it. We substantiate our claim through experiments performed on the simulations and implementations of two dierent distributed systems.

[1]  Debra J. Richardson,et al.  An Analysis of Test Data Selection Criteria Using the RELAY Model of Fault Detection , 1993, IEEE Trans. Software Eng..

[2]  Michael D. Rice,et al.  An approach to architectural analysis and testing , 1998, ISAW '98.

[3]  David R. Karger,et al.  Chord: a scalable peer-to-peer lookup protocol for internet applications , 2003, TNET.

[4]  Paulo César Masiero,et al.  Mutation testing applied to validate specifications based on statecharts , 1999, Proceedings 10th International Symposium on Software Reliability Engineering (Cat. No.PR00443).

[5]  Henry Muccini,et al.  Using software architecture for code testing , 2004, IEEE Transactions on Software Engineering.

[6]  D. Richard Kuhn.  Fault classes and error detection capability of specification-based testing , 1999, TSEM.

[7]  Debra J. Richardson,et al.  Structural specification-based testing: automated support and experimental evaluation , 1999, ESEC/FSE-7.

[8]  Debra J. Richardson,et al.  Approaches to specification-based testing , 1989 .

[9]  Yanyan Wang,et al.  Automating experimentation on distributed testbeds , 2005, ASE.

[10]  Paul Ammann,et al.  A specification-based coverage metric to evaluate test sets , 1999, Proceedings 4th IEEE International Symposium on High-Assurance Systems Engineering.

[11]  Phyllis G. Frankl,et al.  An Experimental Comparison of the Effectiveness of Branch Testing and Data Flow Testing , 1993, IEEE Trans. Software Eng..

[12]  Richard Lai,et al.  A survey of communication protocol testing , 2002, J. Syst. Softw..

[13]  Alexander L. Wolf,et al.  Acm Sigsoft Software Engineering Notes Vol 17 No 4 Foundations for the Study of Software Architecture , 2022 .

[14]  Lionel C. Briand,et al.  Using simulation to empirically investigate test coverage criteria based on statechart , 2004, Proceedings. 26th International Conference on Software Engineering.

[15]  Paul Ammann,et al.  A SPECIFICATION-BASED COVERAGE METRIC TO EVALUATE TEST SETS , 2001 .

[16]  Eric R. Ziegel,et al.  Probability and Statistics for Engineering and the Sciences , 2004, Technometrics.

[17]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[18]  E PerryDewayne,et al.  Foundations for the study of software architecture , 1992 .

[19]  Chengjie Liu,et al.  Maintaining Strong Cache Consistency in the World Wide Web , 1998, IEEE Trans. Computers.

[20]  Elaine J. Weyuker,et al.  An Applicable Family of Data Flow Testing Criteria , 1988, IEEE Trans. Software Eng..

[21]  Simone do Rocio Senger de Souza,et al.  Mutation testing applied to Estelle specifications , 2000, Proceedings of the 33rd Annual Hawaii International Conference on System Sciences.

[22]  Michael D. Ernst,et al.  Improving test suites via operational abstraction , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[23]  Sriram Sankar,et al.  Structural specification-based testing with ADL , 1996, ISSTA '96.

[24]  Phyllis G. Frankl,et al.  Further empirical studies of test effectiveness , 1998, SIGSOFT '98/FSE-6.

[25]  Elaine J. Weyuker,et al.  Automatically Generating Test Data from a Boolean Specification , 1994, IEEE Trans. Software Eng..

[26]  Keith W. Ross,et al.  Computer networking - a top-down approach featuring the internet , 2000 .

[27]  Alexander L. Wolf,et al.  Software testing at the architectural level , 1996, ISAW '96.

[28]  Alexandre Petrenko,et al.  Protocol testing: review of methods and relevance for software testing , 1994, ISSTA '94.

[29]  Shaoying Liu,et al.  Generating test data from SOFL specifications , 1999, J. Syst. Softw..

[30]  A. Jefferson Offutt,et al.  MuJava: an automated class mutation system , 2005, Softw. Test. Verification Reliab..

[31]  A. Jefferson Offutt,et al.  Deriving tests from software architectures , 2001, Proceedings 12th International Symposium on Software Reliability Engineering.

[32]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[33]  Shaoying Liu,et al.  Generating test data from state‐based specifications , 2003, Softw. Test. Verification Reliab..

[34]  Marc J. Balcer,et al.  The category-partition method for specifying and generating fuctional tests , 1988, CACM.

[35]  Ian Clarke,et al.  Protecting Free Expression Online with Freenet , 2002, IEEE Internet Comput..

[36]  Gregg Rothermel,et al.  An experimental determination of sufficient mutant operators , 1996, TSEM.

[37]  Mark Allman,et al.  On the effective evaluation of TCP , 1999, CCRV.