A decision-theoretic approach to the test allocation problem in partition testing

A partition testing strategy consists of two components: a partitioning scheme which determines the way in which the program's input domain is partitioned into subdomains; and an allocation of test cases which determines the exact number of test cases selected from each subdomain. This paper investigates the problem of determining the test allocation when a particular partitioning scheme has been chosen. We show that this problem can be formulated as a classic problem of decision-making under uncertainty, and analyze several well known criteria to resolve this kind of problem. We present algorithms that solve the test allocation problem based on these criteria, and evaluate these criteria by means of a simulation experiment. We also discuss the applicability and implications of applying these criteria in the context of partition testing.

[1]  Jeffrey M. Voas,et al.  Estimating the Probability of Failure When Testing Reveals No Failures , 1992, IEEE Trans. Software Eng..

[2]  Tsong Yueh Chen,et al.  Optimal improvement of the lower bound performance of partition testing strategies , 1997, IEE Proc. Softw. Eng..

[3]  Robert M. Hierons,et al.  Estimation of Failure Rate using Random and Partition Testing , 1997, Softw. Test. Verification Reliab..

[4]  Vijayan N. Nair,et al.  A statistical assessment of some software testing strategies and application of experimental design techniques , 1998 .

[5]  Leonard J. Savage,et al.  The Theory of Statistical Decision , 1951 .

[6]  Walter J. Gutjahr.  Optimal Test Distributions for Software Failure Cost Estimation , 1995, IEEE Trans. Software Eng..

[7]  Tsong Yueh Chen,et al.  On the criteria of allocating test cases under uncertainty , 1997, Proceedings of Joint 4th International Computer Science Conference and 4th Asia Pacific Software Engineering Conference.

[8]  Elaine J. Weyuker,et al.  Analyzing Partition Testing Strategies , 1991, IEEE Trans. Software Eng..

[9]  Elaine J. Weyuker,et al.  A Formal Analysis of the Fault-Detecting Ability of Testing Methods , 1993, IEEE Trans. Software Eng..

[10]  Simeon C. Ntafos,et al.  On some reliability estimation problems in random and partition testing , 1991, Proceedings. 1991 International Symposium on Software Reliability Engineering.

[11]  S. French,et al.  Decision Theory: An Introduction to the Mathematics of Rationality. , 1988 .

[12]  Tsong Yueh Chen,et al.  Constraints for Safe Partition Testing Strategies , 1996, Comput. J..

[13]  Bev Littlewood,et al.  Evaluating Testing Methods by Delivered Reliability , 1998, IEEE Trans. Software Eng..

[14]  Marc J. Balcer,et al.  The category-partition method for specifying and generating fuctional tests , 1988, CACM.

[15]  Tsong Yueh Chen,et al.  On the Expected Number of Failures Detected by Subdomain Testing and Random Testing , 1996, IEEE Trans. Software Eng..

[16]  Tsong Yueh Chen,et al.  On the Relationship Between Partition and Random Testing , 1994, IEEE Trans. Software Eng..

[17]  Simeon C. Ntafos,et al.  An Evaluation of Random Testing , 1984, IEEE Transactions on Software Engineering.

[18]  Tsong Yueh Chen,et al.  Proportional sampling strategy: a compendium and some insights , 2001, J. Syst. Softw..

[19]  Tsong Yueh Chen,et al.  A Study on a Path-based Strategy for Selecting Black-box Generated Test Cases , 2001, Int. J. Softw. Eng. Knowl. Eng..

[20]  Matthias Grochtmann,et al.  Classification trees for partition testing , 1993, Softw. Test. Verification Reliab..

[21]  Richard G. Hamlet,et al.  Partition Testing Does Not Inspire Confidence , 1990, IEEE Trans. Software Eng..

[22]  Elaine J. Weyuker,et al.  Provable Improvements on Branch Testing , 1993, IEEE Trans. Software Eng..

[23]  S. French.  Decision Theory: An Introduction to the Mathematics of Rationality , 1986 .

[24]  Robert M. Hierons.  Testing from a Z Specification , 1997, Softw. Test. Verification Reliab..

[25]  Tsong Yueh Chen,et al.  An Integrated Classification-Tree Methodology for Test Case Generation , 2000, Int. J. Softw. Eng. Knowl. Eng..

[26]  David Gale,et al.  Review: R. Duncan Luce and Howard Raiffa, Games and decisions: Introduction and critical survey , 1958 .

[27]  S. Vajda,et al.  GAMES AND DECISIONS; INTRODUCTION AND CRITICAL SURVEY. , 1958 .

[28]  Dolores R. Wallace,et al.  Enhancing competitiveness via a public fault and failure data repository , 1998, Proceedings Third IEEE International High-Assurance Systems Engineering Symposium (Cat. No.98EX231).

[29]  A. Jefferson Offutt,et al.  An Approach to Fault Modeling and Fault Seeding Using the Program Dependence Graph , 1997, J. Syst. Softw..

[30]  James Miller,et al.  Towards a benchmark for the evaluation of software testing techniques , 1995, Inf. Softw. Technol..

[31]  Tsong Yueh Chen,et al.  On the maximin algorithms for test allocations in partition testing , 2001, Inf. Softw. Technol..