Testing randomized software by means of statistical hypothesis tests

Software testing research has mostly focused on deterministic software systems so far. In reality, however, randomized software systems (i.e. software systems with random output) also play an important role, e. g. for simulation purposes. Test evaluation is a real problem in that case. In previous work, statistical hypothesis tests have already been used, but test decisions have not been interpreted. Furthermore, those tests have only been applied if theoretic values on the distribution of program outputs had been available and not in case of golden implementations. In the present paper, we propose a general approach on how to apply statistical hypothesis tests in order to test randomized software systems. We exactly determine the confidence gained through these tests. We show that after passing a statistical hypothesis test, it can be guaranteed that at least the tested characteristics of the system under test are correct with a certain probability and accuracy of the result. Our approach is also applicable in case of golden implementations. Therefore, knowledge about the outputs' distribution is not necessary in that situation, which is a great advantage. Two case studies are described that have been conducted in order to assess the proposed approach. One of the case studies is based on a software system for the simulation of stochastic geometric models (among others) that evolved from the GeoStoch research project and is now used at France Télécom R&D, Paris, in order to calculate costs for communication networks and to plan new network structures.

[1]  A. Jefferson Offutt,et al.  Mutation 2000: uniting the orthogonal , 2001 .

[2]  Dexter Kozen,et al.  Semantics of probabilistic programs , 1979, 20th Annual Symposium on Foundations of Computer Science (sfcs 1979).

[3]  F. Wilcoxon.  Individual Comparisons by Ranking Methods , 1945 .

[4]  Gregg Rothermel,et al.  A Unifying Framework Supporting the Analysis and Development of Safe Regression Test Selection Techniques , 1999 .

[5]  K. N. King,et al.  A fortran language system for mutation‐based software testing , 1991, Softw. Pract. Exp..

[6]  Robert M. Hierons,et al.  Testing from a nondeterministic finite state machine using adaptive state counting , 2004, IEEE Transactions on Computers.

[7]  Robert V. Binder,et al.  Testing Object-Oriented Systems: Models, Patterns, and Tools , 1999 .

[8]  Volker Schmidt,et al.  Simulation of typical Cox–Voronoi cells with a special regard to implementation tests , 2005, Math. Methods Oper. Res..

[9]  Volker Schmidt,et al.  A unified simulation framework for spatial stochastic models , 2004, Simul. Model. Pract. Theory.

[10]  David Monniaux.  An abstract Monte-Carlo method for the analysis of probabilistic programs , 2001, POPL '01.

[11]  Clemens A. Szyperski,et al.  Component software - beyond object-oriented programming , 2002 .

[12]  Claude Caci,et al.  Testing object-oriented systems , 2000, SOEN.

[13]  Johannes Mayer,et al.  Test Oracles Using Statistical Methods , 2004, SOQUA/TECOS.

[14]  Margus Veanes,et al.  Optimal strategies for testing nondeterministic systems , 2004, ISSTA '04.

[15]  Elaine J. Weyuker,et al.  On Testing Non-Testable Programs , 1982, Comput. J..

[16]  Volker Schmidt,et al.  Fitting of stochastic telecommunication network models via distance measures and Monte–Carlo tests , 2006, Telecommun. Syst..

[17]  David Monniaux.  Abstraction of Expectation Functions Using Gaussian Distributions , 2003, VMCAI.

[18]  Alan Borning,et al.  Automated testing of stochastic systems: a statistically grounded approach , 2006, ISSTA '06.

[19]  Johannes Mayer,et al.  On Testing Image Processing Applications with Statistical Methods , 2005, Software Engineering.

[20]  Joseph Mecke,et al.  Parametric representation of mean values for stationary random mosaies , 1984 .

[21]  V. Schmidt,et al.  Stationary iterated tessellations , 2003, Advances in Applied Probability.

[22]  A. Jefferson Offutt,et al.  MuJava: an automated class mutation system , 2005, Softw. Test. Verification Reliab..

[23]  Herbert Wiklicky,et al.  Probabilistic Abstract Interpretation and Statistical Testing , 2002, PAPM-PROBMIV.