Partition testing, stratified sampling, and cluster analysis

We present a new approach to reducing the manual labor required to estimate software reliability. It combines the ideas of partition testing methods with those of stratified sampling to reduce the sample size necessary to estimate reliability with a given degree of precision. Program executions are stratified by using automatic cluster analysis to group those with similar features. We describe the conditions under which stratification is effective for estimating software reliability, and we present preliminary experimental results suggesting that our approach may work well in practice.

[1]  Ricky W. Butler,et al.  The infeasibility of experimental quantification of life-critical software reliability , 1991 .

[2]  M. Lipow,et al.  Testing for software reliability , 1975, Reliable Software.

[3]  George J. Schick,et al.  An Analysis of Competing Software Reliability Models , 1978, IEEE Transactions on Software Engineering.

[4]  Jeffrey M. Voas,et al.  Estimating the Probability of Failure When Testing Reveals No Failures , 1992, IEEE Trans. Software Eng..

[5]  Elaine J. Weyuker,et al.  Analyzing Partition Testing Strategies , 1991, IEEE Trans. Software Eng..

[6]  I. Olkin,et al.  Sampling theory of surveys, with applications , 1955 .

[7]  Simeon C. Ntafos,et al.  An Evaluation of Random Testing , 1984, IEEE Transactions on Software Engineering.

[8]  Catherine C. McGeoch.  Analyzing algorithms by simulation: variance reduction techniques and simulation speedups , 1992, CSUR.

[9]  Elaine J. Weyuker,et al.  Selecting Software Test Data Using Data Flow Information , 1985, IEEE Transactions on Software Engineering.

[10]  William E. Howden,et al.  Methodology for the Generation of Program Test Data , 1975, IEEE Transactions on Computers.

[11]  Harlan D. Mills,et al.  Certifying the reliability of software , 1986, IEEE Transactions on Software Engineering.

[12]  Bev Littlewood,et al.  THE RISKS OF SOFTWARE , 1992 .

[13]  Joe W. Duran,et al.  Quantifying Software Validity by Sampling , 1980, IEEE Transactions on Reliability.

[14]  J. Neyman.  On the Two Different Aspects of the Representative Method: the Method of Stratified Sampling and the Method of Purposive Selection , 1934 .

[15]  B. Nelson,et al.  A perspective on variance reduction in dynamic simulation experiments , 1987 .

[16]  G. B. Finelli,et al.  The infeasibility of experimental quantification of life-critical software reliability , 1991, SIGSOFT '91.

[17]  Richard G. Hamlet,et al.  Partition Testing Does Not Inspire Confidence , 1990, IEEE Trans. Software Eng..

[18]  P. V. Sukhatme,et al.  Sampling Theory of Surveys with Applications. , 1957 .

[19]  John D. Musa,et al.  Software reliability - measurement, prediction, application , 1987, McGraw-Hill series in software engineering and technology.

[20]  D MusaJohn.  Operational Profiles in Software-Reliability Engineering , 1993 .

[21]  Jon Bentley.  Programming pearls: profilers , 1987, CACM.

[22]  Eldred Nelson,et al.  Estimating software reliability from test data , 1978 .

[23]  John D. Musa,et al.  Operational profiles in software-reliability engineering , 1993, IEEE Software.

[24]  Lori A. Clarke,et al.  A partition analysis method to increase program reliability , 1981, ICSE '81.

[25]  Simeon C. Ntafos,et al.  On some reliability estimation problems in random and partition testing , 1991, Proceedings. 1991 International Symposium on Software Reliability Engineering.

[26]  Donald Ervin Knuth,et al.  The Art of Computer Programming , 1968 .