Experiences and results from initiating field defect prediction and product test prioritization efforts at ABB Inc.

Quantitatively-based risk management can reduce the risks associated with field defects for both software producers and software consumers. In this paper, we report experiences and results from initiating risk-management activities at a large systems development organization. The initiated activities aim to improve product testing (system/integration testing), to improve maintenance resource allocation, and to plan for future process improvements. The experiences we report address practical issues not commonly addressed in research studies: how to select an appropriate modeling method for product testing prioritization and process improvement planning, how to evaluate accuracy of predictions across multiple releases in time, and how to conduct analysis with incomplete information. In addition, we report initial empirical results for two systems with 13 and 15 releases. We present prioritization of configurations to guide product testing, field defect predictions within the first year of deployment to aid maintenance resource allocation, and important predictors across both systems to guide process improvement planning. Our results and experiences are steps towards quantitatively-based risk management.

[1]  N. E. Schneidewind,et al.  Body of Knowledge for Software Quality Measurement , 2002, Computer.

[2]  Hoang Pham.  Software Reliability , 1999 .

[3]  Michael R. Lyu,et al.  What is software reliability? , 1994, Proceedings of COMPASS'94 - 1994 IEEE 9th Annual Conference on Computer Assurance.

[4]  John D. Musa,et al.  Software reliability - measurement, prediction, application , 1987, McGraw-Hill series in software engineering and technology.

[5]  Mary Shaw,et al.  An Empirical Comparison of Field Defect Modeling Methods , 2005 .

[6]  Mayuram S. Krishnan,et al.  Effects of Process Maturity on Quality, Cycle Time, and Effort in Software Product Development , 2000 .

[7]  Audris Mockus,et al.  Predictors of customer perceived software quality , 2005, ICSE.

[8]  Elliot Soloway,et al.  Where the bugs are , 1985, CHI '85.

[9]  K. Fernow.  New York , 1896, American Potato Journal.

[10]  Taghi M. Khoshgoftaar,et al.  Application of neural networks to software quality modeling of a very large telecommunications system , 1997, IEEE Trans. Neural Networks.

[11]  Taghi M. Khoshgoftaar,et al.  Predicting Software Development Errors Using Software Complexity Metrics , 1990, IEEE J. Sel. Areas Commun..

[12]  Francisco D. Galiana,et al.  Unit commitment by simulated annealing , 1990 .

[13]  W. Morton,et al.  Using new packages for modelling, equation oriented simulation and optimization of a cogeneration plant , 2000 .

[14]  Vijay Vittal,et al.  Solution for the crisis in electric power supply , 2001 .

[15]  Per Runeson,et al.  Experience from replicating empirical studies on prediction models , 2002, Proceedings Eighth IEEE Symposium on Software Metrics.

[16]  Abhijit S. Pandya,et al.  A neural network approach for predicting software development faults , 1992, [1992] Proceedings Third International Symposium on Software Reliability Engineering.

[17]  Audris Mockus,et al.  Understanding and predicting effort in software projects , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[18]  Mary Shaw,et al.  Forecasting field defect rates using a combined time-based and metrics-based approach: a case study of OpenBSD , 2005, 16th IEEE International Symposium on Software Reliability Engineering (ISSRE'05).

[19]  S. Weisberg.  Applied Linear Regression, 2nd Edition. , 1987 .

[20]  Taghi M. Khoshgoftaar,et al.  Tree-based software quality estimation models for fault prediction , 2002, Proceedings Eighth IEEE Symposium on Software Metrics.

[21]  Mary Shaw,et al.  Finding predictors of field defects for open source software systems in commonly available data sources: a case study of OpenBSD , 2005, 11th IEEE International Software Metrics Symposium (METRICS'05).

[22]  Taghi M. Khoshgoftaar,et al.  Predicting fault-prone software modules in embedded systems with classification trees , 1999, Proceedings 4th IEEE International Symposium on High-Assurance Systems Engineering.

[23]  Shari Lawrence Pfleeger,et al.  Software metrics (2nd ed.): a rigorous and practical approach , 1997 .

[24]  Mary Shaw,et al.  Empirical evaluation of defect projection models for widely-deployed production software systems , 2004, SIGSOFT '04/FSE-12.

[25]  Norman E. Fenton,et al.  Software metrics: roadmap , 2000, ICSE '00.

[26]  Reynaldo Francisco Nuqui,et al.  State Estimation and Voltage Security Monitoring Using Synchronized Phasor Measurements , 2001 .

[27]  Taghi M. Khoshgoftaar,et al.  A comparative study of predictive models for program changes during system testing and maintenance , 1993, 1993 Conference on Software Maintenance.

[28]  Adam A. Porter,et al.  Learning from Examples: Generation and Evaluation of Decision Trees for Software Resource Analysis , 1988, IEEE Trans. Software Eng..

[29]  G. Q. Kenny.  Estimating defects in commercial software during operational use , 1993 .

[30]  Abhijit S. Pandya,et al.  Application of neural networks for predicting program faults , 1995, Ann. Softw. Eng..

[31]  J. C. Steed.  Aspects of how asset management can be influenced by modern condition monitoring and information management systems , 1998 .

[32]  Taghi M. Khoshgoftaar,et al.  Modeling fault-prone modules of subsystems , 2000, Proceedings 11th International Symposium on Software Reliability Engineering. ISSRE 2000.

[33]  Ignacio E. Grossmann,et al.  A Rigorous MINLP Model for the Optimal Synthesis and Operation of Utility Plants , 1998 .

[34]  H. W. Hale.  A Logic for Identifying the Trees of a Graph , 1961, Transactions of the American Institute of Electrical Engineers. Part III: Power Apparatus and Systems.

[35]  Norman F. Schneidewind,et al.  IEEE Standard For A Software Quality Metrics Methodology Revision And Reaffirmation , 1997, Proceedings of IEEE International Symposium on Software Engineering Standards.

[36]  Taghi M. Khoshgoftaar,et al.  Predicting software errors, during development, using nonlinear regression models: a comparative study , 1992 .

[37]  Taghi M. Khoshgoftaar,et al.  Application of a usage profile in software quality models , 1999, Proceedings of the Third European Conference on Software Maintenance and Reengineering (Cat. No. PR00090).

[38]  Gregory Tassey,et al.  Prepared for what , 2007 .

[39]  Taghi M. Khoshgoftaar,et al.  The dimensionality of program complexity , 1989, ICSE '89.

[40]  Michael R. Lyu,et al.  Handbook of software reliability engineering , 1996 .

[41]  T. Baldwin,et al.  Power system observability with minimal phasor measurement placement , 1993 .

[42]  Emile H. L. Aarts,et al.  Simulated Annealing: Theory and Applications , 1987, Mathematics and Its Applications.

[43]  Martin J. Shepperd.  Evaluating Software Project Prediction Systems , 2005, IEEE METRICS.

[44]  Taghi M. Khoshgoftaar,et al.  Controlling overfitting in software quality models: experiments with regression trees and classification , 2001, Proceedings Seventh International Software Metrics Symposium.

[45]  A. Poeltl,et al.  Experiences with condition monitoring of HV circuit breakers , 2001, 2001 IEEE/PES Transmission and Distribution Conference and Exposition. Developing New Perspectives (Cat. No.01CH37294).