Dynamic test planning: a study in an industrial context

Testing accounts for a relevant part of the production cost of complex or critical software systems. Nevertheless, time and resources budgeted to testing are often underestimated with respect to the target quality goals. Test managers need engineering methods to perform appropriate choices in spending testing resources, so as to maximize the outcome. We present a method to dynamically allocate testing resources to software components minimizing the estimated number of residual defects and/or the estimated residual defect density. We discuss the application to a real-world critical system in the homeland security domain. We describe a support tool aimed at easing industrial technology transfer by hiding to practitioners the mathematical details of the method application.

[1]  Giovanni Denaro,et al.  An empirical evaluation of fault-proneness models , 2002, ICSE '02.

[2]  Amrit L. Goel,et al.  Time-Dependent Error-Detection Rate Model for Software Reliability and Other Performance Measures , 1979, IEEE Transactions on Reliability.

[3]  Domenico Cotroneo,et al.  Testing techniques selection based on ODC fault types and software metrics , 2013, J. Syst. Softw..

[4]  Tadashi Dohi,et al.  An iterative scheme for maximum likelihood estimation in software reliability modeling , 2003, 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003..

[5]  Ramanath Subramanyam,et al.  Empirical Analysis of CK Metrics for Object-Oriented Design Complexity: Implications for Software Defects , 2003, IEEE Trans. Software Eng..

[6]  Domenico Cotroneo,et al.  Combining Operational and Debug Testing for Improving Reliability , 2013, IEEE Transactions on Reliability.

[7]  Tadashi Dohi,et al.  Software reliability growth model with normal distribution and its parameter estimation , 2011, 2011 International Conference on Quality, Reliability, Risk, Maintenance, and Safety Engineering.

[8]  R. N. Rattihalli,et al.  Failure Size Proportional Models and an Analysis of Failure Detection Abilities of Software Testing Strategies , 2007, IEEE Transactions on Reliability.

[9]  Chris F. Kemerer,et al.  A Metrics Suite for Object Oriented Design , 2015, IEEE Trans. Software Eng..

[10]  Michiel van Genuchten,et al.  Using Software Reliability Growth Models in Practice , 2007, IEEE Software.

[11]  CatalCagatay,et al.  A systematic review of software fault prediction studies , 2009 .

[12]  Shigeru Yamada,et al.  Software Reliability Growth Models with Testing-Effort , 1986, IEEE Transactions on Reliability.

[13]  Joanne Bechta Dugan Automated analysis of phased-mission reliability , 1991 .

[14]  Kishor S. Trivedi,et al.  Software Reliability and Testing Time Allocation: An Architecture-Based Approach , 2010, IEEE Transactions on Software Engineering.

[15]  Andreas Zeller,et al.  Mining metrics to predict component failures , 2006, ICSE.

[16]  Shigeru Yamada,et al.  S-Shaped Reliability Growth Modeling for Software Error Detection , 1983, IEEE Transactions on Reliability.

[17]  Niclas Ohlsson,et al.  Predicting Fault-Prone Software Modules in Telephone Switches , 1996, IEEE Trans. Software Eng..

[18]  Tadashi Dohi,et al.  Gompertz software reliability model: Estimation algorithm and empirical validation , 2009, J. Syst. Softw..

[19]  Maurice H. Halstead,et al.  Elements of software science , 1977 .

[20]  Tadashi Dohi,et al.  EM algorithms for logistic software reliability models , 2004, IASTED Conf. on Software Engineering.

[21]  Robert E. Mullen,et al.  The lognormal distribution of software failure rates: application to software reliability growth modeling , 1998, Proceedings Ninth International Symposium on Software Reliability Engineering (Cat. No.98TB100257).

[22]  Kishor S. Trivedi,et al.  Online Monitoring of Software System Reliability , 2010, 2010 European Dependable Computing Conference.

[23]  Michael R. Lyu,et al.  An Assessment of Testing-Effort Dependent Software Reliability Growth Models , 2007, IEEE Transactions on Reliability.

[24]  Banu Diri,et al.  A systematic review of software fault prediction studies , 2009, Expert Syst. Appl..

[25]  Maurice H. Halstead,et al.  Elements of software science (Operating and programming systems series) , 1977 .

[26]  Michael R. Lyu,et al.  Optimal allocation of testing resources for modular software systems , 2002, 13th International Symposium on Software Reliability Engineering, 2002. Proceedings..

[27]  Sinno Jialin Pan,et al.  Transfer defect learning , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[28]  Shigeru Yamada,et al.  Optimal allocation policies for testing-resource based on a software reliability growth model , 1995 .

[29]  Catherine Stringfellow,et al.  An Empirical Method for Selecting Software Reliability Growth Models , 2002, Empirical Software Engineering.

[30]  Sy-Yen Kuo,et al.  Efficient allocation of testing resources for software module testing based on the hyper-geometric distribution software reliability growth model , 1996, Proceedings of ISSRE '96: 7th International Symposium on Software Reliability Engineering.

[31]  Swapna S. Gokhale,et al.  Regression Tree Modeling For The Prediction Of Software Quality , 1997 .

[32]  Victor R. Basili,et al.  A Validation of Object-Oriented Design Metrics as Quality Indicators , 1996, IEEE Trans. Software Eng..

[33]  Amrit L. Goel,et al.  Software Reliability Models: Assumptions, Limitations, and Applicability , 1985, IEEE Transactions on Software Engineering.

[34]  Swapna S. Gokhale,et al.  Log-logistic software reliability growth model , 1998, Proceedings Third IEEE International High-Assurance Systems Engineering Symposium (Cat. No.98EX231).

[35]  Tim Menzies,et al.  Data Mining Static Code Attributes to Learn Defect Predictors , 2007, IEEE Transactions on Software Engineering.

[36]  Elaine J. Weyuker,et al.  Predicting the location and number of faults in large software systems , 2005, IEEE Transactions on Software Engineering.

[37]  Michael R. Lyu,et al.  Handbook of software reliability engineering , 1996 .

[38]  Chin-Yu Huang,et al.  Optimal resource allocation for cost and reliability of modular software systems in the testing phase , 2006, J. Syst. Softw..

[39]  Michael R. Lyu,et al.  Optimal allocation of test resources for software reliability growth modeling in software development , 2002, IEEE Trans. Reliab..

[40]  W. W. Everett,et al.  Software component reliability analysis , 1999, Proceedings 1999 IEEE Symposium on Application-Specific Systems and Software Engineering and Technology. ASSET'99 (Cat. No.PR00122).

[41]  Harald C. Gall,et al.  Cross-project defect prediction: a large scale experiment on data vs. domain vs. process , 2009, ESEC/SIGSOFT FSE.