Reliability estimation for statistical usage testing using Markov chains

Software validation is an important activity in order to test whether or not the correct software has been developed. Several testing techniques have been developed, and one of these is statistical usage testing (SUT). The main purpose of SUT is to test a software product from a user's point of view. Hence, usage models are designed and then test cases are developed from the models. Another advantage of SUT is that the reliability of the software can be estimated. In this paper, Markov chains are used to represent the usage models. Several approaches using Markov chains have been applied. This paper extends these approaches and presents a new approach to estimate the reliability from Markov chains. The reliability estimation is implemented in a new tool for statistical usage testing called MaTeLo. The tool is developed in a joint European project involving six industrial partners and two university partners. The purpose of the tool is to provide an estimate of the reliability and to automatically produce test cases based on usage models described as to Markov models.

[1]  Thomas Thelin,et al.  Practical experiences with statistical usage testing , 2003, Eleventh Annual International Workshop on Software Technology and Engineering Practice.

[2]  Katerina Goseva-Popstojanova,et al.  Architecture-based approach to reliability assessment of software systems , 2001, Perform. Evaluation.

[3]  Robert Oshana,et al.  Improving software quality using statistical testing techniques , 2000, Inf. Softw. Technol..

[4]  John D. Musa,et al.  Software-Reliability-Engineered Testing , 1996, Computer.

[5]  Ara Kouchakdjian,et al.  Improving a product with usage-based testing , 2000, Inf. Softw. Technol..

[6]  John D. Musa,et al.  Operational profiles in software-reliability engineering , 1993, IEEE Software.

[7]  Mark C. Paulk,et al.  Software Product Evaluation , 2001 .

[8]  Harlan D. Mills,et al.  Engineering software under statistical quality control , 1990, IEEE Software.

[9]  Katerina Goseva-Popstojanova,et al.  Assessing uncertainty in reliability of component-based software systems , 2003, 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003..

[10]  Claes Wohlin,et al.  Prioritized Use Cases as a Vehicle for Software Inspections , 2003, IEEE Softw..

[11]  Jesse H. Poore,et al.  Improved techniques for software testing based on markov chain usage models , 1999 .

[12]  James A. Whittaker,et al.  A Markov Chain Model for Statistical Software Testing , 1994, IEEE Trans. Software Eng..

[13]  Roger C. Cheung,et al.  A User-Oriented Software Reliability Model , 1978, IEEE Transactions on Software Engineering.

[14]  Katerina Goseva-Popstojanova,et al.  Software reliability estimation under certainty: generalization of the method of moments , 2004, Eighth IEEE International Symposium on High Assurance Systems Engineering, 2004. Proceedings..

[15]  Walter J. Gutjahr,et al.  Importance Sampling of Test Cases in Markovian Software Usage Models , 1997, Probability in the Engineering and Informational Sciences.

[16]  Edward N. Adams,et al.  Optimizing Preventive Service of Software Products , 1984, IBM J. Res. Dev..

[17]  Claes Wohlin,et al.  An Experimental Comparison of Usage-Based and Checklist-Based Reading , 2003, IEEE Trans. Software Eng..

[18]  Per Runeson,et al.  Derivation of an integrated operational profile and use case model , 1998, Proceedings Ninth International Symposium on Software Reliability Engineering (Cat. No.98TB100257).