The adoption of capture-recapture in software engineering: a systematic literature review

Context: Capture-recapture method has long been adopted in software engineering as a relatively objective way for defect estimation. While many relevant studies have been carried out to evaluate various capture-recapture models and estimators, there still lacks common understanding on the adoption status of the method in software engineering. It is necessary to systematically collect empirical evidence of Capture-recapture adoption hence form necessary understanding on the method. Objective: This study aims to synthesize relevant primary studies on the adoption of capture-recapture method in software engineering, and try to identify possible gaps between the state-of-practice and the state-of-art so as to provide clues for future research. Method: By following the guidelines of Kitchenham, we conducted a Systematic Literature Review(SLR) on studies of the adoption of capture-recapture method in software engineering. Results: From 5 common digital libraries, we retrieved 506 published articles, among them 44 were identified as relevant primary studies. We identified 18 capture-recapture estimators under 4 basic models. Types of the currently existing studies as well as the relevant influencing factors to adoption of the capture-recapture method are also discussed. Conclusion: Results show that there are no conclusive decisions on the best capture-recapture models and estimators. Besides, the number of inspectors and their capability to detect defects as well as the difficulty to detect defects are most critical influencing factors. In addition, lacking of industrial application may be the major issue of current adoption status of capture-recapture method in software engineering.

[1]  Tracy Hall,et al.  A Systematic Literature Review on Fault Prediction Performance in Software Engineering , 2012, IEEE Transactions on Software Engineering.

[2]  Stefan Biffl,et al.  Using Inspection Data for Defect Estimation , 2000, IEEE Softw..

[3]  Nader B. Ebrahimi,et al.  On the Statistical Analysis of the Number of Errors Remaining in a Software Design Document after Inspection , 1997, IEEE Trans. Software Eng..

[4]  Paul S. F. Yip,et al.  An Inference Procedure for a Capture and Recapture Experiment with Time-Dependent Capture Probabilities , 1989 .

[5]  Marc Roper,et al.  Estimating Fault Numbers Remaining After Testing , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[6]  John D. Ferguson,et al.  ASSISTing Management Decisions in the Software Inspection Process , 2002, Inf. Technol. Manag..

[7]  Muhammad Ali Babar,et al.  On Searching Relevant Studies in Software Engineering , 2010, EASE.

[8]  Lionel C. Briand,et al.  A Comprehensive Evaluation of Capture-Recapture Models for Estimating Software Defect Content , 2000, IEEE Trans. Software Eng..

[9]  Paul S. F. Yip,et al.  A Martingale Estimating Equation for a Capture-Recapture Experiment in Discrete Time , 1991 .

[10]  Claes Wohlin,et al.  Capture-recapture in software inspections after 10 years research--theory, evaluation and application , 2004, J. Syst. Softw..

[11]  Yu Hayakawa,et al.  A Gibbs-sampler approach to estimate the number of faults in a system using capture-recapture sampling [software reliability] , 2000, IEEE Trans. Reliab..

[12]  Claes Wohlin,et al.  Defect content estimation for two reviewers , 2001, Proceedings 12th International Symposium on Software Reliability Engineering.

[13]  V. Gupta,et al.  A system for controlling software inspections , 2003, CCECE 2003 - Canadian Conference on Electrical and Computer Engineering. Toward a Caring and Humane Technology (Cat. No.03CH37436).

[14]  Young H. Chun,et al.  Estimating the number of undetected software errors via the correlated capture-recapture model , 2006, Eur. J. Oper. Res..

[15]  Claes Wohlin,et al.  An Experimental Evaluation of an Experience-Based Capture-Recapture Method in Software Code Inspections , 1998, Empirical Software Engineering.

[16]  Anne Chao,et al.  An overview of closed capture-recapture models , 2001 .

[17]  CatalCagatay,et al.  A systematic review of software fault prediction studies , 2009 .

[18]  A. Kamel,et al.  Capture-recapture techniques in software verification , 2008, 2008 International Conference on Computer Engineering & Systems.

[19]  Kenneth H. Pollock,et al.  Modeling capture, recapture, and removal statistics for estimation of demographic parameters for fish and wildlife populations : Past, present, and future , 1991 .

[20]  Lawrence G. Votta,et al.  Assessing Software Designs Using Capture-Recapture Methods , 1993, IEEE Trans. Software Eng..

[21]  Per Runeson,et al.  Robust estimations of fault content with capture-recapture and detection profile estimators , 2000, J. Syst. Softw..

[22]  J. Miller.  On the independence of software inspectors , 2002, J. Syst. Softw..

[23]  Lionel C. Briand,et al.  A comparison and integration of capture-recapture models and the detection profile method , 1998, Proceedings Ninth International Symposium on Software Reliability Engineering (Cat. No.98TB100257).

[24]  David R. Anderson,et al.  Statistical inference from capture data on closed animal populations , 1980 .

[25]  Paul S. F. Yip,et al.  Estimating the number of faults: efficiency of removal, recapture, and seeding , 1999 .

[26]  Lionel C. Briand,et al.  Quantitative evaluation of capture-recapture models to control software inspections , 1997, Proceedings The Eighth International Symposium on Software Reliability Engineering.

[27]  Jeffrey C. Carver,et al.  Application of Kusumoto cost-metric to evaluate the cost effectiveness of software inspections , 2012, Proceedings of the 2012 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement.

[28]  Stefan Biffl,et al.  Investigating the impact of reading techniques on the accuracy of different defect content estimation techniques , 2001, Proceedings Seventh International Software Metrics Symposium.

[29]  Stephen G. Eick,et al.  Estimating software fault content before coding , 1992, International Conference on Software Engineering.

[30]  Sadahiro Isoda,et al.  A criticism on the capture-and-recapture method for software reliability assurance , 1995, Proceedings 1995 Asia Pacific Software Engineering Conference.

[31]  Cagatay Catal,et al.  Software fault prediction: A literature review and current trends , 2011, Expert Syst. Appl..

[32]  Anne Chao,et al.  Estimating Animal Abundance with Capture Frequency Data , 1988 .

[33]  Stefan Biffl,et al.  Evaluating defect estimation models with major defects , 2003, J. Syst. Softw..

[34]  Claes Wohlin,et al.  Evaluating Defect Content Estimation Rules in Software Inspections , 2000 .

[35]  Thomas Thelin,et al.  Team-based fault content estimation in the software inspection process , 2004, Proceedings. 26th International Conference on Software Engineering.

[36]  Claes Wohlin,et al.  Defect content estimations from review data , 1998, Proceedings of the 20th International Conference on Software Engineering.

[37]  Tsvi Kuflik,et al.  Estimating the Required Code Inspection Team Size , 2007, IEEE International Conference on Software-Science, Technology & Engineering (SwSTE'07).

[38]  A Chao,et al.  Estimating population size via sample coverage for closed capture-recapture models. , 1994, Biometrics.

[39]  Frank Padberg.  Empirical interval estimates for the defect content after an inspection , 2002, ICSE '02.

[40]  Claes Wohlin,et al.  An empirical study of experience-based software defect content estimation methods , 1999, Proceedings 10th International Symposium on Software Reliability Engineering (Cat. No.PR00443).

[41]  David R. Anderson,et al.  Capture-Recapture and Removal Methods for Sampling Closed Populations , 1983 .

[42]  Claes Wohlin,et al.  Capture-recapture in software unit testing: a case study , 2008, ESEM '08.

[43]  Stefan Biffl,et al.  Evaluating the accuracy of defect estimation models based on inspection data from two inspection cycles , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[44]  Claes Wohlin,et al.  Evaluation of using Capture-Recapture Methods on Software Review Data , 1999 .

[45]  Per Runeson,et al.  Confidence intervals for capture-recapture estimations in software inspections , 2002, Inf. Softw. Technol..

[46]  D. G. Chapman.  Some properties of the hypergeometric distribution with applications to zoölogical somple censuses , 1951 .

[47]  Jeffrey C. Carver,et al.  Evaluation of capture-recapture models for estimating the abundance of naturally-occurring defects , 2008, ESEM '08.

[48]  James Miller.  Estimating the number of remaining defects after inspection , 1999 .

[49]  Chih-Ping Chu,et al.  A defect estimation approach for sequential inspection using a modified capture-recapture model , 2005, 29th Annual International Computer Software and Applications Conference (COMPSAC'05).

[50]  Phillip A. Laplante,et al.  A dynamic capture–recapture model for software defect prediction , 2009, Innovations in Systems and Software Engineering.

[51]  Claes Wohlin,et al.  Estimating the number of components with defects post‐release that showed no defects in testing , 2002, Softw. Test. Verification Reliab..

[52]  K. Burnham,et al.  Estimation of the size of a closed population when capture probabilities vary among animals , 1978 .

[53]  J. Darroch.  THE MULTIPLE-RECAPTURE CENSUS I. ESTIMATION OF A CLOSED POPULATION , 1958 .

[54]  Jeffrey C. Carver,et al.  The effect of the number of inspectors on the defect estimates produced by capture-recapture models , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[55]  A. Chao.  Estimating the population size for capture-recapture data with unequal catchability. , 1987, Biometrics.

[56]  Jeffrey C. Carver,et al.  Evaluating the Effect of the Number of Naturally Occurring Faults on the Estimates Produced by Capture-Recapture Models , 2009, 2009 International Conference on Software Testing Verification and Validation.

[57]  Thomas Thelin,et al.  Empirical Evaluations of Usage-Based Reading and Fault Content Estimation for Software Inspections , 2003, Empirical Software Engineering.

[58]  Banu Diri,et al.  A systematic review of software fault prediction studies , 2009, Expert Syst. Appl..

[59]  Khaled El Emam,et al.  Evaluating Capture-Recapture Models with Two Inspectors , 2001, IEEE Trans. Software Eng..

[60]  Paul G. Sorenson,et al.  The application of capture-recapture log-linear models to software inspections data , 2003, 2003 International Symposium on Empirical Software Engineering, 2003. ISESE 2003. Proceedings..

[61]  Pearl Brereton,et al.  Performing systematic literature reviews in software engineering , 2006, ICSE.