Application of Kusumoto cost-metric to evaluate the cost effectiveness of software inspections

Inspections and testing are two widely recommended techniques for improving software quality. While testing cannot be conducted until software is implemented, inspections can help find and fix the faults right after their injection in the requirements and design documents. It is estimated that majority of testing cost is spent on fault rework and can be saved by inspections of early software products. However there is a lack of evidence regarding the testing costs saved by performing inspections. This research analyzes the costs and benefits of inspections and testing to decide on whether to schedule an inspection. We also analyzed the effect of the team size on the decision of how to organize the inspections. Another aspect of our research evaluates the use of Capture Recapture (CR) estimation method when the actual fault count of software product is unknown. Using data from 73 inspectors, we applied the Kusumoto metric to evaluate the cost-effectiveness of the inspections with varying team size. Our results provide a detailed analysis of the number of inspectors required for varying levels of cost-effectiveness during inspections; and the number of inspectors required by the CR estimators to provide estimates within 5% to 20% of the actual.

[1]  Anne Chao,et al.  Estimating Animal Abundance with Capture Frequency Data , 1988 .

[2]  Claes Wohlin,et al.  Capture-recapture in software inspections after 10 years research--theory, evaluation and application , 2004, J. Syst. Softw..

[3]  Barry W. Boehm,et al.  Software Defect Reduction Top 10 List , 2001, Computer.

[4]  Michael E. Fagan Advances in software inspections , 1986, IEEE Transactions on Software Engineering.

[5]  Bill Brykczynski,et al.  A survey of software inspection checklists , 1999, SOEN.

[6]  Shinji Kusumoto,et al.  A New Metric for Cost Effectiveness of Software Reviews , 1992 .

[7]  K. Burnham,et al.  Estimation of the size of a closed population when capture probabilities vary among animals , 1978 .

[8]  J. Darroch THE MULTIPLE-RECAPTURE CENSUS I. ESTIMATION OF A CLOSED POPULATION , 1958 .

[9]  Michael E. Fagan Design and Code Inspections to Reduce Errors in Program Development , 1976, IBM Syst. J..

[10]  Barry W. Boehm,et al.  Software Engineering Economics , 1993, IEEE Transactions on Software Engineering.

[11]  Lionel C. Briand,et al.  Using simulation to build inspection efficiency benchmarks for development projects , 1998, Proceedings of the 20th International Conference on Software Engineering.

[12]  David R. Anderson,et al.  Statistical inference from capture data on closed animal populations , 1980 .

[13]  David R. Anderson,et al.  Statistical inference from capture data on closed animal populations , 1980 .

[14]  John C. Kelly,et al.  An analysis of defect densities found during software inspections , 1992, J. Syst. Softw..

[15]  Jeffrey C. Carver The Impact of Background and Experience on Software Inspections , 2004, Empirical Software Engineering.

[16]  Thomas McGibbon,et al.  A Business Case for Software Process Improvement (2007 Update) Measuring Return on Investment from Software Engineering and Management , 2007 .

[17]  James Miller,et al.  Estimating the number of remaining defects after inspection , 1999, Softw. Test. Verification Reliab..

[18]  A Chao,et al.  Estimating population size via sample coverage for closed capture-recapture models. , 1994, Biometrics.

[19]  David R. Anderson,et al.  Capture-Recapture and Removal Methods for Sampling Closed Populations , 1983 .

[20]  Jeffrey C. Carver,et al.  The effect of the number of inspectors on the defect estimates produced by capture-recapture models , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[21]  A. Chao Estimating the population size for capture-recapture data with unequal catchability. , 1987, Biometrics.

[22]  Jeffrey C. Carver,et al.  Using error abstraction and classification to improve requirement quality: conclusions from a family of four empirical studies , 2012, Empirical Software Engineering.

[23]  Khaled El Emam,et al.  The application of subjective estimates of effectiveness to controlling software inspections , 2000, J. Syst. Softw..

[24]  David R. Anderson,et al.  Capture-Recapture and Removal Methods for Sampling Closed Populations , 1983 .

[25]  Barry Boehm,et al.  Top 10 list [software development] , 2001 .

[26]  Jeffrey C. Carver,et al.  Evaluation of capture-recapture models for estimating the abundance of naturally-occurring defects , 2008, ESEM '08.

[27]  Lionel C. Briand,et al.  A comparison and integration of capture-recapture models and the detection profile method , 1998, Proceedings Ninth International Symposium on Software Reliability Engineering (Cat. No.98TB100257).

[28]  Glenford J. Myers,et al.  A controlled experiment in program testing and code walkthroughs/inspections , 1978, CACM.

[29]  A. Frank Ackerman,et al.  Software inspections: an effective verification process , 1989, IEEE Software.

[30]  Sabaliauskaite Giedre,et al.  Investigating defect detection in object-oriented design and cost-effectiveness of software inspection , 2004 .

[31]  Edward F. Weller,et al.  Lessons from three years of inspection data (software development) , 1993, IEEE Software.

[32]  Stefan Biffl,et al.  Investigating the cost-effectiveness of reinspections in software development , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[33]  Peter J. Middleton,et al.  Software Inspection , 1994, J. Inf. Technol..

[34]  Lionel C. Briand,et al.  A Comprehensive Evaluation of Capture-Recapture Models for Estimating Software Defect Content , 2000, IEEE Trans. Software Eng..

[35]  Karl E. Wiegers,et al.  Peer Reviews in Software: A Practical Guide , 2001 .

[36]  Miguel Toro,et al.  A controlled experiment for evaluating a metric-based reading technique for requirements inspection , 2004 .

[37]  Paul S. F. Yip,et al.  A Martingale Estimating Equation for a Capture-Recapture Experiment in Discrete Time , 1991 .

[38]  Stephen G. Eick,et al.  Estimating software fault content before coding , 1992, International Conference on Software Engineering.

[39]  Scott N. Woodfield,et al.  Evaluating the effectiveness of reliability-assurance techniques , 1989, J. Syst. Softw..