Investigating code review quality: Do people and participation matter?

Code review is an essential element of any mature software development project; it aims at evaluating code contributions submitted by developers. In principle, code review should improve the quality of code changes (patches) before they are committed to the project's master repository. In practice, bugs are sometimes unwittingly introduced during this process. In this paper, we report on an empirical study investigating code review quality for Mozilla, a large open-source project. We explore the relationships between the reviewers' code inspections and a set of factors, both personal and social in nature, that might affect the quality of such inspections. We applied the SZZ algorithm to detect bug-inducing changes that were then linked to the code review information extracted from the issue tracking system. We found that 54% of the reviewed changes introduced bugs in the code. Our findings also showed that both personal metrics, such as reviewer workload and experience, and participation metrics, such as the number of involved developers, are associated with the quality of the code review process.

[1]  N. Nagappan,et al.  Use of relative code churn measures to predict system defect density , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[2]  Witold Pedrycz,et al.  A comparative analysis of the efficiency of change metrics and static code attributes for defect prediction , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[3]  Eric S. Raymond,et al.  The cathedral and the bazaar - musings on Linux and Open Source by an accidental revolutionary , 2001 .

[4]  Victor R. Basili,et al.  The influence of organizational structure on software quality , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[5]  Bret Tilson.  Modules , 2010, Int. J. Algebra Comput..

[6]  Yi Zhang,et al.  Classifying Software Changes: Clean or Buggy? , 2008, IEEE Transactions on Software Engineering.

[7]  Harald C. Gall,et al.  Don't touch my code!: examining the effects of ownership on software quality , 2011, ESEC/FSE '11.

[8]  Audris Mockus,et al.  Software Dependencies, Work Dependencies, and Their Impact on Failures , 2009, IEEE Transactions on Software Engineering.

[9]  Michael W. Godfrey,et al.  The Secret Life of Patches: A Firefox Case Study , 2012, 2012 19th Working Conference on Reverse Engineering.

[10]  Akito Monden,et al.  An analysis of developer metrics for fault prediction , 2010, PROMISE '10.

[11]  Lin Tan,et al.  Do time of day and developer experience affect commit bugginess? , 2011, MSR '11.

[12]  Harvey P. Siy,et al.  Predicting Fault Incidence Using Software Change History , 2000, IEEE Trans. Software Eng..

[13]  Daniel M. Germán,et al.  Will my patch make it? And how fast? Case study on the Linux kernel , 2013, 2013 10th Working Conference on Mining Software Repositories (MSR).

[14]  Rainer Koschke,et al.  Effort-Aware Defect Prediction Models , 2010, 2010 14th European Conference on Software Maintenance and Reengineering.

[15]  Thomas Zimmermann,et al.  Automatic Identification of Bug-Introducing Changes , 2006, 21st IEEE/ACM International Conference on Automated Software Engineering (ASE'06).

[16]  Andy Zaidman,et al.  Modern code reviews in open-source projects: which problems do they fix? , 2014, MSR 2014.

[17]  Michael W. Godfrey,et al.  The influence of non-technical factors on code review , 2013, 2013 20th Working Conference on Reverse Engineering (WCRE).

[18]  Foutse Khomh,et al.  Do code review practices impact design quality? A case study of the Qt, VTK, and ITK projects , 2015, 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER).

[19]  Michael E. Fagan.  Design and Code Inspections to Reduce Errors in Program Development , 1976, IBM Syst. J..

[20]  Premkumar T. Devanbu,et al.  Ownership, experience and defects: a fine-grained study of authorship , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[21]  J. Fox,et al.  Applied Regression Analysis and Generalized Linear Models , 2008 .

[22]  Andreas Zeller,et al.  When do changes induce fixes? , 2005, ACM SIGSOFT Softw. Eng. Notes.

[23]  Alberto Bacchelli,et al.  Expectations, outcomes, and challenges of modern code review , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[24]  Mark C. Paulk,et al.  The Impact of Design and Code Reviews on Software Quality: An Empirical Study Based on PSP Data , 2009, IEEE Transactions on Software Engineering.

[25]  Ahmed E. Hassan,et al.  Predicting faults using the complexity of code changes , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[26]  Margaret-Anne D. Storey,et al.  Understanding broadcast based peer review on open source software projects , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[27]  Michael Fagan.  Design and Code Inspections to Reduce Errors in Program Development , 1976, IBM Syst. J..

[28]  Stephan Diehl,et al.  Small patches get in! , 2008, MSR '08.

[29]  Harald C. Gall,et al.  Cross-project defect prediction: a large scale experiment on data vs. domain vs. process , 2009, ESEC/SIGSOFT FSE.

[30]  Audris Mockus,et al.  A large-scale empirical study of just-in-time quality assurance , 2013, IEEE Transactions on Software Engineering.

[31]  Mika Mäntylä,et al.  What Types of Defects Are Really Discovered in Code Reviews? , 2009, IEEE Transactions on Software Engineering.

[32]  J. Fox.  Applied Regression Analysis, Linear Models, and Related Methods , 1997 .

[33]  M. Uihlein.  Open , 2018 .

[34]  Shane McIntosh,et al.  The impact of code review coverage and code review participation on software quality: a case study of the qt, VTK, and ITK projects , 2014, MSR 2014.

[35]  Jacob Cohen,et al.  Applied multiple regression/correlation analysis for the behavioral sciences , 1979 .

[36]  Gabriele Bavota,et al.  When Does a Refactoring Induce Bugs? An Empirical Study , 2012, 2012 IEEE 12th International Working Conference on Source Code Analysis and Manipulation.

[37]  Audris Mockus,et al.  Organizational volatility and its effects on software defects , 2010, FSE '10.

[38]  Les Hatton,et al.  Testing the Value of Checklists in Code Inspections , 2008, IEEE Software.