A time/structure based software reliability model

The past 20 years have seen the formulation of numerous analytical software reliability models for estimating the reliability growth of a software product. The predictions obtained by applying these models tend to be optimistic due to the inaccuracies in the operational profile, and saturation effect of testing. Incorporating knowledge gained about some structural attribute of the code, such as test coverage, into the time-domain models can help alleviate this optimistic trend. In this paper we present an enhanced non-homogeneous Poisson process (ENHPP) model which incorporates explicitly the time-varying test-coverage function in its analytical formulation, and provides for defective fault detection and test coverage during the testing and operational phases. It also allows for a time varying fault detection rate. The ENHPP model offers a unifying framework for all the previously reported finite failure NHPP models via test coverage. We also propose the log-logistic coverage function which can capture an increasing/decreasing failure detection rate per fault, which cannot be accounted for by the previously reported finite failure NHPP models. We present a methodology based on the ENHPP model for reliability prediction earlier in the testing phase. Expressions for predictions in the operational phase of the software, software availability, and optimal software release times subject to various constraints such as cost, reliability, and availability are developed based on the ENHPP model. We also validate the ENHPP model based on four different coverage functions using five failure data sets.

[1]  Jean-Claude Laprie,et al.  Trend Analysis , 1996, The SAGE Encyclopedia of Research Design.

[2]  William Farr,et al.  Software reliability modeling survey , 1996 .

[3]  Karama Kanoun,et al.  A Method for Software Reliability Analysis and Prediction Application to the TROPICO-R Switching System , 1991, IEEE Trans. Software Eng..

[4]  Michael R. Lyu,et al.  CASRE: a computer-aided software reliability estimation tool , 1992, [1992] Proceedings of the Fifth International Workshop on Computer-Aided Software Engineering.

[5]  Jean-Claude Laprie,et al.  Qualitative and Quantitative Reliability Assessment , 1997, IEEE Softw..

[6]  Vernon Rego,et al.  A case study to investigate sensitivity of reliability estimates to errors in operational profile , 1994, Proceedings of 1994 IEEE International Symposium on Software Reliability Engineering.

[7]  J. R. Horgan,et al.  A data flow coverage testing tool for C , 1992, [1992] Proceedings of the Second Symposium on Assessment of Quality Software Development Tools.

[8]  Claes Wohlin,et al.  A practical method for the estimation of software reliability growth in the early stage of testing , 1997, Proceedings The Eighth International Symposium on Software Reliability Engineering.

[9]  W. H. Mac Williams Keynote address , 2006, AIEE-IRE '51.

[10]  Paul Piwowarski,et al.  Coverage measurement experience during function test , 1993, Proceedings of 1993 15th International Conference on Software Engineering.

[11]  William E. Howden,et al.  Functional Program Testing , 1978, IEEE Transactions on Software Engineering.

[12]  Mitsuru Ohba,et al.  Software Reliability Analysis Models , 1984, IBM J. Res. Dev..

[13]  John D. Musa,et al.  The operational profile , 1996 .

[14]  Jean-Claude Laprie,et al.  Software Reliability Trend Analyses from Theoretical to Practical Considerations , 1994, IEEE Trans. Software Eng..

[15]  EhrlichWilla,et al.  Determining the Cost of a Stop-Test Decision , 1993 .

[16]  Raymond Jacoby,et al.  Test coverage dependent software reliability estimation by the HGD model , 1992, [1992] Proceedings Third International Symposium on Software Reliability Engineering.

[17]  Shunji Osaki,et al.  Software Reliability Growth Modeling: Models and Applications , 1985, IEEE Transactions on Software Engineering.

[18]  Vernon Rego,et al.  Effect of testing techniques on software reliability estimates obtained using a time-domain model , 1995 .

[19]  S. A. Hossain,et al.  Estimating the parameters of a non-homogeneous Poisson-process model for software reliability , 1993 .

[20]  Trevor Hastie,et al.  Statistical Models in S , 1991 .

[21]  Joseph Robert Horgan,et al.  Data flow coverage and the C language , 1991, TAV4.

[22]  Carlo Batini,et al.  A layout algorithm for data flow diagrams , 1986, IEEE Transactions on Software Engineering.

[23]  Joseph A. Morgan,et al.  Solving ML equations for 2-parameter Poisson-process models for ungrouped software-failure data , 1996, IEEE Trans. Reliab..

[24]  James M. Bieman,et al.  The relationship between test coverage and reliability , 1994, Proceedings of 1994 IEEE International Symposium on Software Reliability Engineering.

[25]  G. Q. Kenny Estimating defects in commercial software during operational use , 1993 .

[26]  Olivier Gaudoin,et al.  Optimal properties of the Laplace trend test for soft-reliability models , 1992 .

[27]  Barry W. Boehm,et al.  A spiral model of software development and enhancement , 1986, Computer.

[28]  Joseph Robert Horgan,et al.  Effect of test set size and block coverage on the fault detection effectiveness , 1994, Proceedings of 1994 IEEE International Symposium on Software Reliability Engineering.

[29]  Boudewijn R. Haverkort,et al.  Performance and reliability analysis of computer systems: An example-based approach using the sharpe software package , 1998 .

[30]  Vernon Rego,et al.  Perils Of Software Reliability Modeling , 1995 .

[31]  M. L. Shooman,et al.  Software reliability: A historical perspective , 1984, IEEE Transactions on Reliability.

[32]  Sy-Yen Kuo,et al.  Optimal Release Times for Software Systems with Scheduled Delivery Time Based on the HGDM , 1997, IEEE Trans. Computers.

[33]  Elaine J. Weyuker,et al.  Selecting Software Test Data Using Data Flow Information , 1985, IEEE Transactions on Software Engineering.

[34]  D MusaJohn Operational Profiles in Software-Reliability Engineering , 1993 .

[35]  Amrit L. Goel,et al.  Time-Dependent Error-Detection Rate Model for Software Reliability and Other Performance Measures , 1979, IEEE Transactions on Reliability.

[36]  Jim Gray,et al.  Why Do Computers Stop and What Can Be Done About It? , 1986, Symposium on Reliability in Distributed Software and Database Systems.

[37]  Shigeru Yamada,et al.  S-Shaped Reliability Growth Modeling for Software Error Detection , 1983, IEEE Transactions on Reliability.

[38]  Joseph Robert Horgan,et al.  Effect of Test Set Minimization on Fault Detection Effectiveness , 1995, 1995 17th International Conference on Software Engineering.

[39]  Michael R. Lyu,et al.  Incorporating code coverage in the reliability estimation for fault-tolerant software , 1997, Proceedings of SRDS'97: 16th IEEE Symposium on Reliable Distributed Systems.

[40]  Sarah Brocklehurst,et al.  Techniques for prediction analysis and recalibration , 1996 .

[41]  Swapna S. Gokhale,et al.  Important Milestones in Software Reliability Modeling , 1996, SEKE.

[42]  Sarah Brocklehurst,et al.  New Ways to Get Accurate Reliability Measures , 1992, IEEE Softw..

[43]  Willa K. Ehrlich,et al.  Determining the cost of a stop-test decision (software reliability) , 1993, IEEE Software.

[44]  Aditya P. Mathur,et al.  Software testing and reliability , 1996 .

[45]  Bev Littlewood,et al.  Evaluation of competing software reliability predictions , 1986, IEEE Transactions on Software Engineering.

[46]  Amrit L. Goel,et al.  Software Reliability Models: Assumptions, Limitations, and Applicability , 1985, IEEE Transactions on Software Engineering.

[47]  Kishor S. Trivedi,et al.  Performance And Reliability Analysis Of Computer Systems (an Example-based Approach Using The Sharpe Software , 1997, IEEE Transactions on Reliability.

[48]  Alberto Pasquini,et al.  Sensitivity of reliability-growth models to operational profile errors vs. testing accuracy [software testing] , 1996, IEEE Trans. Reliab..

[49]  Joseph Robert Horgan,et al.  Assessing testing tools in research and education , 1992, IEEE Software.

[50]  David R. Cox,et al.  The statistical analysis of series of events , 1966 .

[51]  Fabio Del Frate,et al.  On the correlation between code coverage and software reliability , 1995, Proceedings of Sixth International Symposium on Software Reliability Engineering. ISSRE'95.

[52]  Siddhartha R. Dalal,et al.  Buying with Exact Confidence , 1992 .

[53]  Sheldon M. Ross,et al.  Software Reliability: The Stopping Rule Problem , 1985, IEEE Transactions on Software Engineering.

[54]  C. Mallows,et al.  When Should One Stop Testing Software , 1988 .

[55]  M. Rosenblatt Remarks on a Multivariate Transformation , 1952 .

[56]  C. V. Ramamoorthy,et al.  Software Reliability—Status and Perspectives , 1982, IEEE Transactions on Software Engineering.

[57]  Michael R. Lyu,et al.  Handbook of software reliability engineering , 1996 .

[58]  Kishor S. Trivedi Probability and Statistics with Reliability, Queuing, and Computer Science Applications , 1984 .

[59]  Michael R. Lyu,et al.  An empirical study of the correlation between code coverage and reliability estimation , 1996, Proceedings of the 3rd International Software Metrics Symposium.

[60]  Michael R. Lyu,et al.  A coverage analysis tool for the effectiveness of software testing , 1993, Proceedings of 1993 IEEE International Symposium on Software Reliability Engineering.

[61]  Michael R. Lyu,et al.  Achieving software quality with testing coverage measures , 1994, Computer.

[62]  KuoSy-Yen,et al.  Optimal Release Times for Software Systems with Scheduled Delivery Time Based on the HGDM , 1997 .

[63]  Harry Joe,et al.  Statistical Inference for General-Order-Statistics and Nonhomogeneous-Poisson-Process Software Reliability Models , 1989, IEEE Trans. Software Eng..

[64]  John E. Gaffney,et al.  On predicting software reliability , 1989, [1989] Proceedings of the Thirteenth Annual International Computer Software & Applications Conference.

[65]  John D. Musa,et al.  Operational profiles in software-reliability engineering , 1993, IEEE Software.

[66]  Victor R. Basili,et al.  Analyzing the test process using structural coverage , 1985, ICSE '85.

[67]  Michael R. Lyu,et al.  An experiment in determining software reliability model applicability , 1995, Proceedings of Sixth International Symposium on Software Reliability Engineering. ISSRE'95.