A Comparative Analysis of Software Reliability Growth Models using Defects Data of Closed and Open Source Software

The purpose of this study is to compare the fitting (goodness of fit) and prediction capability of eight Software Reliability Growth Models (SRGM) using fifty different failure Data sets. These data sets contain defect data collected from system test phase, operational phase (field defects) and Open Source Software (OSS) projects. The failure data are modelled by eight SRGM (Musa Okumoto, Inflection S-Shaped, Goel Okumoto, Delayed S-Shaped, Logistic, Gompertz, Yamada Exponential, and Generalized Goel Model). These models are chosen due to their prevalence among many software reliability models. The results can be summarized as follows: Fitting capability: Musa Okumoto fits all data sets, but all models fit all the OSS datasets. : Prediction capability: Musa Okumoto, Inflection S-Shaped and Goel Okumoto are the best predictors for industrial data sets, Gompertz and Yamada are the best predictors for OSS data sets. : Fitting and prediction capability: Musa Okumoto and Inflection are the best performers on industrial datasets. However this happens only on slightly more than 50% of the datasets. Gompertz and Inflection are the best performers for all OSS datasets.

[1]  Taghi M. Khoshgoftaar,et al.  Software reliability model selection: a cast study , 1991, Proceedings. 1991 International Symposium on Software Reliability Engineering.

[2]  Carina Andersson,et al.  A replicated empirical study of a selection method for software reliability growth models , 2007, Empirical Software Engineering.

[3]  Chin-Yu Huang,et al.  An integration of fault detection and correction processes in software reliability analysis , 2006, J. Syst. Softw..

[4]  Xiaolin Teng,et al.  Considering fault removal efficiency in software reliability assessment , 2003, IEEE Trans. Syst. Man Cybern. Part A.

[5]  Eoca Raton.  SOFTWARE RELIABILITY MODEL SELECTION: A CASE STUDY , 1991 .

[6]  Richard P. Heydorn,et al.  Reliability Engineering Handbook , 2001, Technometrics.

[7]  Michael R. Lyu,et al.  Handbook of software reliability engineering , 1996 .

[8]  Giancarlo Succi,et al.  Modelling Failures Occurrences of Open Source Software with Reliability Growth , 2010, OSS.

[9]  Xuemei Zhang,et al.  Adjusting software failure rates that are estimated from test data , 2005, IEEE Transactions on Reliability.

[10]  John E. Gaffney,et al.  On predicting software reliability , 1989, [1989] Proceedings of the Thirteenth Annual International Computer Software & Applications Conference.

[11]  Alan Wood,et al.  Software reliability growth models: assumptions vs. reality , 1997, Proceedings The Eighth International Symposium on Software Reliability Engineering.

[12]  Ying Zhou,et al.  Open source software reliability model , 2005, ACM SIGSOFT Softw. Eng. Notes.

[13]  Shigeru Yamada,et al.  Software Reliability Growth Models with Testing-Effort , 1986, IEEE Transactions on Reliability.

[14]  Xuemei Zhang,et al.  Some successful approaches to software reliability modeling in industry , 2005, J. Syst. Softw..

[15]  Azad H. Azadmanesh,et al.  A Comparative Analysis of Open Source Software Reliability , 2010, J. Softw..

[16]  Joseph G. Davis,et al.  Analyzing and Modeling Open Source Software Bug Report Data , 2008, 19th Australian Conference on Software Engineering (aswec 2008).

[17]  Santosh Kumar,et al.  Software-reliability growth model: primary-failures generate secondary-faults under imperfect debugging , 1994 .

[18]  Xuemei Zhang,et al.  Accounting for realities when estimating the field failure rate of software , 2001, Proceedings 12th International Symposium on Software Reliability Engineering.

[19]  Huifang Cheng,et al.  Improved Genetic Programming Algorithm Applied to Symbolic Regression and Software Reliability Modeling , 2009, J. Softw. Eng. Appl..

[20]  AnderssonCarina.  A replicated empirical study of a selection method for software reliability growth models , 2007 .

[21]  John D. Musa,et al.  Software reliability measurement , 1984, J. Syst. Softw..

[22]  Catherine Stringfellow,et al.  An Empirical Method for Selecting Software Reliability Growth Models , 2002, Empirical Software Engineering.

[23]  Taghi M. Khoshgoftaar,et al.  Software reliability model selection , 1992 .

[24]  M. Trachtenberg.  A general theory of software-reliability modeling , 1990 .

[25]  Brendan Murphy,et al.  Post-release reliability growth in software products , 2008, TSEM.

[26]  Kapil Sharma,et al.  Selection of Optimal Software Reliability Growth Models Using a Distance Based Approach , 2010, IEEE Transactions on Reliability.

[27]  John D. Musa,et al.  Quantifying Software Validation: When to Stop Testing? , 1989, IEEE Softw..

[28]  Shigeru Yamada,et al.  S-Shaped Reliability Growth Modeling for Software Error Detection , 1983, IEEE Transactions on Reliability.

[29]  Tom McBride,et al.  Reliability Growth of Open Source Software Using Defect Analysis , 2008, 2008 International Conference on Computer Science and Software Engineering.

[30]  Joseph G. Davis,et al.  Analyzing and Modeling Open Source Software Bug Report Data , 2008 .

[31]  Amrit L. Goel,et al.  Time-Dependent Error-Detection Rate Model for Software Reliability and Other Performance Measures , 1979, IEEE Transactions on Reliability.

[32]  Yeu-Shiang Huang,et al.  A study of software reliability growth from the perspective of learning effects , 2008, Reliab. Eng. Syst. Saf..

[33]  M. Akber Qureshi,et al.  Estimating the failure rate of evolving software systems , 2000, Proceedings 11th International Symposium on Software Reliability Engineering. ISSRE 2000.

[34]  Sarah Brocklehurst,et al.  Recalibrating Software Reliability Models , 1990, IEEE Trans. Software Eng..

[35]  H. Derriennic,et al.  Use of failure-intensity models in the software-validation phase for telecommunications , 1995 .

[36]  K.B.P.L.M. Kelani Bandara,et al.  Optimal selection of failure data for reliability estimation based on a standard deviation method , 2007, 2007 International Conference on Industrial and Information Systems.

[37]  H. D. Rombach,et al.  The Goal Question Metric Approach , 1994 .

[38]  Michael R. Lyu,et al.  A Unified Scheme of Some Nonhomogenous Poisson Process Models for Software Reliability Estimation , 2003, IEEE Trans. Software Eng..

[39]  Bev Littlewood,et al.  Evaluation of competing software reliability predictions , 1986, IEEE Transactions on Software Engineering.

[40]  Amrit L. Goel,et al.  Software Reliability Models: Assumptions, Limitations, and Applicability , 1985, IEEE Transactions on Software Engineering.

[41]  Min Xie,et al.  A simple goodness-of-fit test for the power-law process, based on the Duane plot , 2003, IEEE Trans. Reliab..

[42]  Qingxin Zhu,et al.  The Software Failure Prediction Based on Fractal , 2008, 2008 Advanced Software Engineering and Its Applications.

[43]  John D. Musa,et al.  Software reliability - measurement, prediction, application , 1987, McGraw-Hill series in software engineering and technology.

[44]  Suku Nair,et al.  A Model for Software Development Effort and Cost Estimation , 1997, IEEE Trans. Software Eng..

[45]  Alan N. Sukert.  Empirical Validation of Three Software Error Prediction Models , 1979, IEEE Transactions on Reliability.

[46]  George J. Schick,et al.  An Analysis of Competing Software Reliability Models , 1978, IEEE Transactions on Software Engineering.

[47]  Xiang Li,et al.  Reliability analysis and optimal version-updating for open source software , 2011, Inf. Softw. Technol..