The Impact of Design and Code Reviews on Software Quality: An Empirical Study Based on PSP Data

This research investigates the effect of review rate on defect removal effectiveness and the quality of software products, while controlling for a number of potential confounding factors. Two data sets of 371 and 246 programs, respectively, from a personal software process (PSP) approach were analyzed using both regression and mixed models. Review activities in the PSP process are those steps performed by the developer in a traditional inspection process. The results show that the PSP review rate is a significant factor affecting defect removal effectiveness, even after accounting for developer ability and other significant process variables. The recommended review rate of 200 LOC/hour or less was found to be an effective rate for individual reviews, identifying nearly two-thirds of the defects in design reviews and more than half of the defects in code reviews.

[1]  Sarah Brocklehurst,et al.  Techniques for prediction analysis and recalibration , 1996 .

[2]  Philip M. Johnson,et al.  The Personal Software Process: A Cautionary Case Study , 1998, IEEE Softw..

[3]  Martha M. Gardner.  Implementing Six Sigma: Smarter Solutions Using Statistical Methods , 2000, Technometrics.

[4]  Fumio Akiyama,et al.  An Example of Software System Debugging , 1971, IFIP Congress.

[5]  Carol Withrow,et al.  Error density and size in Ada software , 1990, IEEE Software.

[6]  S. B. Kiselev,et al.  The capability maturity model: guidelines for improving the software process , 1995 .

[7]  Richard F. Gunst,et al.  Applied Regression Analysis , 1999, Technometrics.

[8]  Chris F. Kemerer,et al.  Cyclomatic Complexity Density and Software Maintenance Productivity , 1991, IEEE Trans. Software Eng..

[9]  R. Littell.  SAS System for Mixed Models , 1996 .

[10]  Claes Wohlin.  Are Individual Differences in Software Development Performance Possible to Capture Using a Quantitative Survey? , 2004, Empirical Software Engineering.

[11]  Sarah Brocklehurst,et al.  Recalibrating Software Reliability Models , 1990, IEEE Trans. Software Eng..

[12]  Harvey P. Siy,et al.  Understanding the Effects of Developer Activities on Inspection Interval , 1997, Proceedings of the (19th) International Conference on Software Engineering.

[13]  Mark C. Paulk,et al.  An empirical study of process discipline and software quality , 2005 .

[14]  Anders Wesslén,et al.  A Replicated Empirical Study of the Impact of the Methods in the PSP on Individual Engineers , 2000, Empirical Software Engineering.

[15]  N. Draper,et al.  Applied Regression Analysis , 1966 .

[16]  David Lorge Parnas,et al.  Active design reviews: principles and practices , 1985, ICSE '85.

[17]  Robert L. Glass.  Practical programmer: inspections—some surprising findings , 1999, CACM.

[18]  V. Barnett,et al.  Applied Linear Statistical Models , 1975 .

[19]  Michael R. Lyu,et al.  Handbook of software reliability engineering , 1996 .

[20]  Bill Curtis,et al.  A field study of the software design process for large systems , 1988, CACM.

[21]  Adam A. Porter,et al.  Anywhere, Anytime Code Inspections: Using the Web to Remove Inspection Bottlenecks in Large-Scale Software Development , 1997, Proceedings of the (19th) International Conference on Software Engineering.

[22]  Edward F. Weller,et al.  Lessons from three years of inspection data (software development) , 1993, IEEE Software.

[23]  Lesley Pek Wee Land.  Software Group Reviews and the Impact of Procedural Roles on Defect Detection Performance , 2004, Empirical Software Engineering.

[24]  Michael E. Fagan.  Advances in software inspections , 1986, IEEE Transactions on Software Engineering.

[25]  Mark C. Paulk,et al.  The Evolation of the SEI's Capacity Maturity Model for Software , 1995, Softw. Process. Improv. Pract..

[26]  Dayanand N. Naik,et al.  Applied Multivariate Statistics with SAS Software , 1997 .

[27]  Harvey P. Siy,et al.  An experiment to assess the cost-benefits of code inspections in large scale software development , 1995, SIGSOFT '95.

[28]  C. Wohlin,et al.  Understanding software defect detection in the Personal Software Process , 1998, Proceedings Ninth International Symposium on Software Reliability Engineering (Cat. No.98TB100257).

[29]  Terry E. Duncan,et al.  An Introduction to Latent Variable Growth Curve Modeling: Concepts, Issues, and Application, Second Edition , 1999 .

[30]  Thomas Gilb,et al.  Software Inspection , 1994 .

[31]  Watts S. Humphrey,et al.  Introduction to the Team Software Process , 1999 .

[32]  Stephen G. Eick,et al.  Estimating software fault content before coding , 1992, International Conference on Software Engineering.

[33]  W. W. Muir,et al.  Regression Diagnostics: Identifying Influential Data and Sources of Collinearity , 1980 .

[34]  W. Edwards Deming,et al.  Out of the Crisis , 1982 .

[35]  Adam A. Porter,et al.  Reducing Inspection Interval in Large-Scale Software Development , 2002, IEEE Trans. Software Eng..

[36]  Watts S. Humphrey,et al.  Results of Applying the Personal Software Process , 1997, Computer.

[37]  Norman E. Fenton,et al.  A Critique of Software Defect Prediction Models , 1999, IEEE Trans. Software Eng..

[38]  A. Frank Ackerman,et al.  Software inspections: an effective verification process , 1989, IEEE Software.

[39]  Lutz Prechelt,et al.  An Experiment Measuring the Effects of Personal Software Process (PSP) Training , 2001, IEEE Trans. Software Eng..

[40]  Rajiv D. Banker,et al.  A model to evaluate variables impacting the productivity of software maintenance projects , 1991 .

[41]  Ronald A. Radice,et al.  High Quality Low Cost Software Inspections , 2001 .

[42]  Michael E. Fagan.  Design and Code Inspections to Reduce Errors in Program Development , 1976, IBM Syst. J..

[43]  Audris Mockus,et al.  Understanding the sources of variation in software inspections , 1998, TSEM.

[44]  D. A. Christenson,et al.  A code inspection model for software quality management and prediction , 1988, IEEE Global Telecommunications Conference and Exhibition. Communications for the Information Age.

[45]  Adam A. Porter,et al.  Assessing Software Review Meetings: Results of a Comparative Analysis of Two Experimental Studies , 1997, IEEE Trans. Software Eng..

[46]  Claes Wohlin,et al.  Defect content estimations from review data , 1998, Proceedings of the 20th International Conference on Software Engineering.

[47]  Ellis Horowitz,et al.  Software Cost Estimation with COCOMO II , 2000 .

[48]  Watts S. Humphrey,et al.  A discipline for software engineering , 2012, Series in software engineering.

[49]  William Hayes,et al.  The Personal Software Process (PSPSM): An Empirical Study of the Impact of PSP on Individual Engineers. , 1997 .

[50]  Muneo Takahashi,et al.  An empirical study of a model for program error prediction , 1985, ICSE '85.

[51]  Robert L. Glass,et al.  Inspections - Some Surprising Findings. , 1999 .