Predicting the Reliability of Mass-Market Software in the Marketplace Based on Beta Usage: a Study of Windows Vista and WIndows 7

The traditional approach for predicting post-release software reliability is to emulate customer usage environments and scenarios during in-house testing. However, this approach has limitations for mass-market software systems (MMSS), such as the Windows operating system, because it is impractical to emulate the usage environments and scenarios of millions of users. This paper presents and validates Usage Profile-based Reliability Measurement Calibration (UPRMC), a novel approach for producing accurate post-release reliability predictions using data from beta releases. This work leverages usage and reliability data from multiple releases of a large commercial operating system spanning more than 3 years and more than 3 million users. Results show that UPRMC produces accurate and credible predictions; it has been adopted by the Windows organization.

[1]  Norman F. Schneidewind,et al.  Applying reliability models to the space shuttle , 1992, IEEE Software.

[2]  Anil K. Jain,et al.  Data clustering: a review , 1999, CSUR.

[3]  Song Xue,et al.  Reliability Assessment of Mass-Market Software: Insights from Windows Vista® , 2008, 2008 19th International Symposium on Software Reliability Engineering (ISSRE).

[4]  M. Akber Qureshi,et al.  Estimating the failure rate of evolving software systems , 2000, Proceedings 11th International Symposium on Software Reliability Engineering. ISSRE 2000.

[5]  Archana Ganapathi,et al.  Windows XP Kernel Crash Analysis , 2006, LISA.

[6]  Mark C. Paulk,et al.  The Capability Maturity Model: Guidelines for Improving the Software Process , 1994 .

[7]  Lori L. Pollock,et al.  Automated Oracle Comparators for TestingWeb Applications , 2007, The 18th IEEE International Symposium on Software Reliability (ISSRE '07).

[8]  Shari Lawrence Pfleeger,et al.  Software Metrics : A Rigorous and Practical Approach , 1998 .

[9]  Alan P. Wood,et al.  Software Reliability from the Customer View , 2003, Computer.

[10]  Paul Luo Li,et al.  Estimating the Quality of Widely Used Software Products Using Software Reliability Growth Modeling: Case Study of an IBM Federated Database Project , 2007, First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007).

[11]  Michael R. Lyu,et al.  Handbook of software reliability engineering , 1996 .

[12]  Martin Höst,et al.  Sensitivity of Website Reliability to Usage Profile Changes , 2007, The 18th IEEE International Symposium on Software Reliability (ISSRE '07).

[13]  Jim Gray,et al.  Why Do Computers Stop and What Can Be Done About It? , 1986, Symposium on Reliability in Distributed Software and Database Systems.

[14]  Dennis R. Goldenson,et al.  A systematic survey of CMM experience and results , 1996, Proceedings of IEEE 18th International Conference on Software Engineering.

[15]  Madeline Diep,et al.  Profiling deployed software: assessing strategies and testing opportunities , 2005, IEEE Transactions on Software Engineering.

[16]  John D. Musa,et al.  Software reliability - measurement, prediction, application , 1987, McGraw-Hill series in software engineering and technology.

[17]  Paul Luo Li,et al.  A Catalog of Techniques that Predict Information about the Count or Rate of Field Defects , 2006 .

[18]  Brendan Murphy.  Automating Software Failure Reporting , 2004, ACM Queue.

[19]  Katerina Goseva-Popstojanova,et al.  Empirical Characterization of Session–Based Workload and Reliability for Web Servers , 2006, Empirical Software Engineering.

[20]  Andreas Zeller,et al.  Mining metrics to predict component failures , 2006, ICSE.

[21]  Mark Russinovich,et al.  Windows® Internals: Including Windows Server 2008 and Windows Vista, Fifth Edition , 2009 .

[22]  Brendan Murphy,et al.  Measuring system and software reliability using an automated data collection process , 1995 .

[23]  P. L. Li,et al.  Estimating the Quality of Widely Used Software Products Using Software Reliability Growth Modeling: Case Study of an IBM Federated Database Project , 2007, ESEM 2007.

[24]  Audris Mockus,et al.  Predictors of customer perceived software quality , 2005, ICSE.