Empirical Evaluations of Usage-Based Reading and Fault Content Estimation for Software Inspections

This thesis describes six empirical investigations of two techniques applied to software inspections, namely reading techniques and fault content estimation techniques. The first part of the thesis presents a series of experiments of a novel reading technique called usage-based reading. The second part investigates fault content estimation applied to the software inspection process. The estimation methods used are capture-recapture and curve fitting.

[1]  A. Chao,et al.  Reliability-estimation and stopping-rules for software testing, based on repeated appearances of bugs , 1995 .

[2]  Forrest Shull,et al.  How perspective-based reading can improve requirements inspections , 2000, Computer.

[3]  Claes Wohlin,et al.  An empirical study of experience-based software defect content estimation methods , 1999, Proceedings 10th International Symposium on Software Reliability Engineering (Cat. No.PR00443).

[4]  David Grover,et al.  Information and software technology : a project-based approach , 2004 .

[5]  David B. Bisant,et al.  Cognitive processes in software fault detection: A review and synthesis , 1993, Int. J. Hum. Comput. Interact..

[6]  Per Runeson,et al.  Are the Perspectives Really Different? – Further Experimentation on Scenario-Based Reading of Requirements , 2000, Empirical Software Engineering.

[7]  John D. Musa,et al.  Software reliability engineering : more reliable software, faster development and testing , 1999 .

[8]  Lionel C. Briand,et al.  Quantitative evaluation of capture-recapture models to control software inspections , 1997, Proceedings The Eighth International Symposium on Software Reliability Engineering.

[9]  Adam A. Porter,et al.  Comparing Detection Methods for Software Requirements Inspections: A Replicated Experiment , 1995, IEEE Trans. Software Eng..

[10]  E. P. Doolan,et al.  Experience with Fagan's inspection method , 1992, Softw. Pract. Exp..

[11]  Stephen G. Eick,et al.  Estimating software fault content before coding , 1992, International Conference on Software Engineering.

[12]  Shari Lawrence Pfleeger,et al.  Software Metrics : A Rigorous and Practical Approach , 1998 .

[13]  Alessandro Bianchi,et al.  A controlled experiment to assess the effectiveness of inspection meetings , 2001, Proceedings Seventh International Software Metrics Symposium.

[14]  Victor R. Basili,et al.  The role of experimentation in software engineering: past, current, and future , 1996, Proceedings of IEEE 18th International Conference on Software Engineering.

[15]  Adam A. Porter,et al.  Comparing Detection Methods For Software Requirements Inspections: A Replication Using Professional Subjects , 1998, Empirical Software Engineering.

[16]  Claes Wohlin,et al.  How much information is needed for usage-based reading? A series of experiments , 2002, Proceedings International Symposium on Empirical Software Engineering.

[17]  Kent L. Beck,et al.  Embracing Change with Extreme Programming , 1999, Computer.

[18]  Walter F. Tichy,et al.  Hints for Reviewing Empirical Work in Software Engineering , 2000, Empirical Software Engineering.

[19]  W. W. Royce,et al.  Managing the development of large software systems , 1970 .

[20]  Frank Padberg Empirical interval estimates for the defect content after an inspection , 2002, ICSE '02.

[21]  Claes Wohlin,et al.  An Experimental Comparison of Usage-Based and Checklist-Based Reading , 2003, IEEE Trans. Software Eng..

[22]  Joachim Karlsson,et al.  A Cost-Value Approach for Prioritizing Requirements , 1997, IEEE Softw..

[23]  R.C. Linger,et al.  Cleanroom process model , 1994, IEEE Software.

[24]  John C. Knight,et al.  An improved inspection technique , 1993, CACM.

[25]  Forrest Shull,et al.  Detecting defects in object-oriented designs: using reading techniques to increase software quality , 1999, OOPSLA '99.

[26]  S. Siegel,et al.  Nonparametric Statistics for the Behavioral Sciences. , 1957 .

[27]  A. Frank Ackerman,et al.  Software inspections: an effective verification process , 1989, IEEE Software.

[28]  Per Runeson A new software engineering programme - structure and initial experiences , 2000, Thirteenth Conference on Software Engineering Education and Training.

[29]  William E. Riddle,et al.  Software technology maturation , 1985, ICSE '85.

[30]  Per Runeson,et al.  Derivation of an integrated operational profile and use case model , 1998, Proceedings Ninth International Symposium on Software Reliability Engineering (Cat. No.98TB100257).

[31]  Colin Atkinson,et al.  An experimental comparison of reading techniques for defect detection in UML design documents , 2000, J. Syst. Softw..

[32]  Michael E. Fagan Design and Code Inspections to Reduce Errors in Program Development , 1976, IBM Syst. J..

[33]  Nader B. Ebrahimi,et al.  On the Statistical Analysis of the Number of Errors Remaining in a Software Design Document after Inspection , 1997, IEEE Trans. Software Eng..

[34]  Edward F. Weller,et al.  Lessons from three years of inspection data (software development) , 1993, IEEE Software.

[35]  Claes Wohlin,et al.  Evaluating Defect Content Estimation Rules in Software Inspections , 2000 .

[36]  Claes Wohlin,et al.  An Experimental Evaluation of an Experience-Based Capture-Recapture Method in Software Code Inspections , 1998, Empirical Software Engineering.

[37]  Ben Shneiderman,et al.  Perspective-based Usability Inspection: An Empirical Validation of Efficacy , 1999, Empirical Software Engineering.

[38]  Marc Roper,et al.  Further investigations into the development and evaluation of reading techniques for object-oriented code inspection , 2002, ICSE '02.

[39]  John D. Ferguson,et al.  ASSISTing Management Decisions in the Software Inspection Process , 2002, Inf. Technol. Manag..

[40]  Matthias Jarke,et al.  Scenarios in System Development: Current Practice , 1998, IEEE Softw..

[41]  Watts S. Humphrey,et al.  A discipline for software engineering , 2012, Series in software engineering.

[42]  Kenneth H. Pollock,et al.  Modeling capture, recapture, and removal statistics for estimation of demographic parameters for fish and wildlife populations : Past, present, and future , 1991 .

[43]  Forrest Shull,et al.  Developing techniques for using software documents: a series of empirical studies , 1998 .

[44]  Wei-Tek Tsai,et al.  N-Fold inspection: a requirements analysis technique , 1990, Commun. ACM.

[45]  William C. Hetzel,et al.  The complete guide to software testing , 1984 .

[46]  John D. Barrett,et al.  Quality From Customer Needs to Customer Satisfaction , 2004, Technometrics.

[47]  Robert G. Ebenau,et al.  Software Inspection Process , 1993 .

[48]  Elaine M. Hall,et al.  Managing Risk: Methods for Software Systems Development , 1998 .

[49]  Lawrence G. Votta,et al.  Assessing Software Designs Using Capture-Recapture Methods , 1993, IEEE Trans. Software Eng..

[50]  Stefan Biffl,et al.  Investigating the impact of reading techniques on the accuracy of different defect content estimation techniques , 2001, Proceedings Seventh International Software Metrics Symposium.

[51]  J. Miller On the independence of software inspectors , 2002, J. Syst. Softw..

[52]  Paul Clements,et al.  Software architecture in practice , 1999, SEI series in software engineering.

[53]  Claes Wohlin,et al.  Experimentation in software engineering: an introduction , 2000 .

[54]  Claes Wohlin,et al.  Capture-recapture in software inspections after 10 years research--theory, evaluation and application , 2004, J. Syst. Softw..

[55]  Claes Wohlin,et al.  Defect content estimation for two reviewers , 2001, Proceedings 12th International Symposium on Software Reliability Engineering.

[56]  Forrest Shull,et al.  Building Knowledge through Families of Experiments , 1999, IEEE Trans. Software Eng..

[57]  Claes Wohlin,et al.  An experimental evaluation of capture‐recapture in software inspections , 1995, Softw. Test. Verification Reliab..

[58]  Michael E. Fagan Advances in software inspections , 1986, IEEE Transactions on Software Engineering.

[59]  Paul Clements,et al.  Software Architecture in Practice: Addison-Wesley , 1998 .

[60]  Oliver Laitenberger,et al.  Perspective-based reading of code documents at Robert Bosch GmbH , 1997, Inf. Softw. Technol..

[61]  James Miller,et al.  Applying meta-analytical procedures to software engineering experiments , 2000, J. Syst. Softw..

[62]  T. Saaty,et al.  The Analytic Hierarchy Process , 1985 .

[63]  Giuseppe Visaggio,et al.  Evaluating Defect Detection Techniques for Software Requirements Inspections , 2000 .

[64]  Susan Wiedenbeck,et al.  What do novices learn during program comprehension? , 1991, Int. J. Hum. Comput. Interact..

[65]  Claes Wohlin,et al.  Using Students as Subjects—A Comparative Study of Students and Professionals in Lead-Time Impact Assessment , 2000, Empirical Software Engineering.

[66]  Lionel C. Briand,et al.  A Comprehensive Evaluation of Capture-Recapture Models for Estimating Software Defect Content , 2000, IEEE Trans. Software Eng..

[67]  Per Runeson,et al.  Robust estimations of fault content with capture-recapture and detection profile estimators , 2000, J. Syst. Softw..

[68]  Barry W. Boehm,et al.  A spiral model of software development and enhancement , 1986, Computer.

[69]  Carothers Ad An examination and extension of Leslie's test of equal catchability. , 1971 .

[70]  Khaled El Emam,et al.  The application of subjective estimates of effectiveness to controlling software inspections , 2000, J. Syst. Softw..

[71]  A D Carothers An examination and extension of Leslie's test of equal catchability. , 1971, Biometrics.

[72]  Oliver Laitenberger,et al.  An encompassing life cycle centric survey of software inspection , 2000, J. Syst. Softw..

[73]  David R. Anderson,et al.  Capture-Recapture and Removal Methods for Sampling Closed Populations , 1983 .

[74]  Lionel C. Briand,et al.  A comparison and integration of capture-recapture models and the detection profile method , 1998, Proceedings Ninth International Symposium on Software Reliability Engineering (Cat. No.98TB100257).

[75]  James D. Nichols,et al.  Capture-recapture models. , 1992 .

[76]  Giuseppe Visaggio,et al.  A Replicated Experiment to Assess Requirements Inspection Techniques , 2004, Empirical Software Engineering.

[77]  A. Chao Estimating the population size for capture-recapture data with unequal catchability. , 1987, Biometrics.

[78]  Stefan Biffl,et al.  Evaluating the accuracy of defect estimation models based on inspection data from two inspection cycles , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[79]  Sivert Sørumgård,et al.  Verification of Process Conformance in Empirical Studies of Software Development , 1997 .

[80]  Claes Wohlin,et al.  Evaluation of using Capture-Recapture Methods on Software Review Data , 1999 .

[81]  Claes Wohlin,et al.  State‐of‐the‐art: software inspections after 25 years , 2002, Softw. Test. Verification Reliab..

[82]  Victor R. Basili,et al.  Evolving and packaging reading technologies , 1997, J. Syst. Softw..

[83]  Anne Chao,et al.  Estimating population size for sparse data in capture-recapture experiments , 1989 .

[84]  C. Wohlin,et al.  Sample-Driven Inspections , 1999 .

[85]  Harlan D. Mills,et al.  Structured programming - theory and practice , 1979, The systems programming series.

[86]  James R. Lyle,et al.  A Two-Person Inspection Method to Improve Prog ramming Productivity , 1989, IEEE Transactions on Software Engineering.

[87]  S. T. Buckland,et al.  An Introduction to the Bootstrap. , 1994 .

[88]  Will Venters,et al.  Software engineering: theory and practice , 2006 .

[89]  Natalia Juristo Juzgado,et al.  Basics of Software Engineering Experimentation , 2010, Springer US.

[90]  David R. Anderson,et al.  Statistical inference from capture data on closed animal populations , 1980 .

[91]  M. Patton,et al.  Qualitative evaluation and research methods , 1992 .

[92]  Margaret J. Robertson,et al.  Design and Analysis of Experiments , 2006, Handbook of statistics.

[93]  James Miller Estimating the number of remaining defects after inspection , 1999 .

[94]  Colin Atkinson,et al.  Generalizing perspective-based inspection to handle object-oriented development artifacts , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[95]  Lesley Pek Wee Land Software Group Reviews and the Impact of Procedural Roles on Defect Detection Performance , 2004, Empirical Software Engineering.

[96]  Fraser Macdonald,et al.  Computer Supported Software Inspection , 1999 .

[97]  Forrest Shull,et al.  The empirical investigation of Perspective-Based Reading , 1995, Empirical Software Engineering.

[98]  Khaled El Emam,et al.  Evaluating Capture-Recapture Models with Two Inspectors , 2001, IEEE Trans. Software Eng..

[99]  Martha E. Crosby,et al.  How do we read algorithms? A case study , 1990, Computer.

[100]  Ivar Jacobson,et al.  Object-oriented software engineering - a use case driven approach , 1993, TOOLS.

[101]  A Chao,et al.  Estimating population size for capture-recapture data when capture probabilities vary by time and individual animal. , 1992, Biometrics.

[102]  Claes Wohlin,et al.  Estimating the number of components with defects post‐release that showed no defects in testing , 2002, Softw. Test. Verification Reliab..

[103]  K. Burnham,et al.  Estimation of the size of a closed population when capture probabilities vary among animals , 1978 .

[104]  D. Ross Jeffery,et al.  Comparing inspection strategies for software requirement specifications , 1996, Proceedings of 1996 Australian Software Engineering Conference.

[105]  Itu-T Specification and Description Language (SDL) , 1999 .

[106]  Soren Lauesen,et al.  Software Requirements: Styles & Techniques , 2002 .

[107]  Stefan Biffl,et al.  Using Inspection Data for Defect Estimation , 2000, IEEE Softw..

[108]  Philip M. Johnson,et al.  Does Every Inspection Really Need a Meeting? , 1998, Empirical Software Engineering.

[109]  Jesse H. Poore,et al.  Bringing Software Under Statistical Quality Control , 1988 .

[110]  John J. Marciniak Reviews and Audits , 2002 .

[111]  David R. Anderson,et al.  Capture-Recapture and Removal Methods for Sampling Closed Populations , 1983 .

[112]  James Miller,et al.  Further Experiences with Scenarios and Checklists , 1998, Empirical Software Engineering.

[113]  David Lorge Parnas,et al.  Active design reviews: principles and practices , 1985, ICSE '85.

[114]  Stefan Biffl,et al.  Using reading techniques to focus inspection performance , 2001, Proceedings 27th EUROMICRO Conference. 2001: A Net Odyssey.

[115]  Lawrence G. Votta,et al.  Does every inspection need a meeting? , 1993, SIGSOFT '93.

[116]  Harvey P. Siy,et al.  An Experiment ot Assess the Cost-Benefits of Code Inspections in Large Scale Software Development , 1997, IEEE Trans. Software Eng..

[117]  Walter F. Tichy,et al.  Should Computer Scientists Experiment More? , 1998, Computer.

[118]  Will Hayes,et al.  Research synthesis in software engineering: a case for meta-analysis , 1999, Proceedings Sixth International Software Metrics Symposium (Cat. No.PR00403).

[119]  Alistair Cockburn,et al.  Writing Effective Use Cases , 2000 .

[120]  B BisantDavid,et al.  A Two-Person Inspection Method to Improve Programming Productivity , 1989 .

[121]  James Miller,et al.  A Comparison of Computer Support Systems for Software Inspection , 2004, Automated Software Engineering.

[122]  Per Runeson,et al.  Usage-based readingan experiment to guide reviewers with use cases , 2001, Inf. Softw. Technol..

[123]  Ola Blomkvist,et al.  An Extended Replication of an Experiment for Assessing Methods for Software Requirements Inspections , 1998, Empirical Software Engineering.

[124]  James Miller,et al.  An empirical evaluation of defect detection techniques , 1997, Inf. Softw. Technol..

[125]  Claes Wohlin,et al.  Statistical Usage Testing for Software Reliability Control , 1995, Informatica.

[126]  Per Runeson,et al.  Confidence intervals for capture-recapture estimations in software inspections , 2002, Inf. Softw. Technol..

[127]  Adam A. Porter,et al.  An experiment to assess different defect detection methods for software requirements inspections , 1994, Proceedings of 16th International Conference on Software Engineering.