Empirical estimates of software availability of deployed systems

We consider empirical evaluation of the availability of the deployed software. Evaluation of real systems is more realistic, more accurate, and provides higher level of confidence than simulations, testing, or models. We process and model information gathered from a variety of operational and service support systems to obtain estimates of software reliability and availability. The three principal quantities are the total runtime, the number of outages, and the duration of outages. We consider methods to assess the quality of information in customer support systems, discuss advantages and disadvantages of various sources, consider methods to deal with missing data, and ways to construct bounds on measures that are not directly available. We propose a method to assess empirically software availability and reliability based on information from operational customer support and inventory systems and use a case study of a large communications system to investigate factors affecting software reliability. We find large variations among platforms and releases and find the failure rate to vary over time.

[1]  Ming-Wei Lu,et al.  Automotive reliability prediction based on early field failure warranty data , 1998 .

[2]  Audris Mockus,et al.  An empirical study of global software development: distance and speed , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[3]  Hoang Pham.  Software Reliability , 1999 .

[4]  David P. Harrington,et al.  Nonparametric estimation of the survival distribution in censored data , 1984 .

[5]  Hans Jörg Wingender.  Reliability Data Collection and Use in Risk and Availability Assessment , 1986 .

[6]  Audris Mockus,et al.  Predicting risk of software changes , 2000, Bell Labs Technical Journal.

[7]  Hoang Pham,et al.  Predicting operational software availability and its applications to telecommunication systems , 2002, Int. J. Syst. Sci..

[8]  C. Kjærgaard.  Field Failure Data Collection and Analysis of Repairable Systems , 1989 .

[9]  Audris Mockus,et al.  Analogy based prediction of work item flow in software projects: a case study , 2003, 2003 International Symposium on Empirical Software Engineering, 2003. ISESE 2003. Proceedings..

[10]  Ping Zhang,et al.  Predictors of customer perceived software quality , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[11]  Audris Mockus,et al.  Expertise Browser: a quantitative approach to identifying expertise , 2002, Proceedings of the 24th International Conference on Software Engineering. ICSE 2002.

[12]  Nanua Singh,et al.  Hazard rate estimation from incomplete and unclean warranty data , 2003, Reliab. Eng. Syst. Saf..

[13]  D. W. Coit,et al.  Practical reliability data and analysis , 1986 .

[14]  Audris Mockus,et al.  Formulation and preliminary test of an empirical theory of coordination in software engineering , 2003, ESEC/FSE-11.

[15]  Audris Mockus,et al.  Identifying reasons for software changes using historic databases , 2000, Proceedings 2000 International Conference on Software Maintenance.

[16]  J. Herbsleb,et al.  Two case studies of open source software development: Apache and Mozilla , 2002, TSEM.

[17]  David W. Coit,et al.  Analysis of grouped data from field-failure reporting systems , 1999 .

[18]  Audris Mockus,et al.  Refactoring for changeability: a way to go? , 2005, 11th IEEE International Software Metrics Symposium (METRICS'05).

[19]  Michael R. Lyu,et al.  What is software reliability? , 1994, Proceedings of COMPASS'94 - 1994 IEEE 9th Annual Conference on Computer Assurance.

[20]  Audris Mockus,et al.  Globalization by Chunking: A Quantitative Approach , 2001, IEEE Softw..

[21]  Audris Mockus,et al.  Measuring technology effects on software change cost , 2000, Bell Labs Technical Journal.

[22]  Audris Mockus,et al.  Drivers for Customer Perceived Software Quality , 2005, ICSE 2005.

[23]  J. P. Fahy.  Estimating warranty and service costs from MTBF estimates , 1995, Proceedings of Electro/International 1995.

[24]  E. Kaplan,et al.  Nonparametric Estimation from Incomplete Observations , 1958 .

[25]  H. Müller,et al.  Hazard rate estimation under random censoring with varying kernels and bandwidths. , 1994, Biometrics.

[26]  Ke Liu.  Refrigerator failure early prediction based on warranty data , 2002, Annual Reliability and Maintainability Symposium. 2002 Proceedings (Cat. No.02CH37318).

[27]  H. A. Malec,et al.  Communications reliability: a historical perspective , 1998 .

[28]  Audris Mockus,et al.  Understanding and predicting effort in software projects , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[29]  Audris Mockus,et al.  Using Version Control Data to Evaluate the Impact of Software Tools: A Case Study of the Version Editor , 2002, IEEE Trans. Software Eng..

[30]  Jean-Claude Laprie,et al.  Experience in software reliability: From data collection to quantitative evaluation , 1993, Proceedings of 1993 IEEE International Symposium on Software Reliability Engineering.

[31]  S. K. Kurtz,et al.  A simple method for predicting the cumulative failures of consumer products during the warranty period , 1995, Annual Reliability and Maintainability Symposium 1995 Proceedings.