Automatically recommending code reviewers based on their expertise: An empirical comparison

Code reviews are an essential part of quality assurance in Free, Libre, and Open Source Software (FLOSS) projects. However, finding a suitable reviewer can be difficult, and delayed or forgotten reviews are the consequence. Automating reviewer selection with suitable algorithms can mitigate this problem. We compare empirically six algorithms based on modification expertise and two algorithms based on review expertise on four major FLOSS projects. Our results indicate that the algorithms based on review expertise yield better recommendations than those based on modification expertise. The algorithm Weighted Review Count (WRC) recommends at least one out of five reviewers correctly in 69 % to 75 % of all cases, which is one of the best results achieved in the comparison.

[1]  Bhuricha Deen Sethanandha,et al.  On the Need for OSS Patch Contribution Tools , 2010 .

[2]  A. L. Edwards.  Note on the “correction for continuity” in testing the significance of the difference between correlated proportions , 1948, Psychometrika.

[3]  Michael W. Godfrey,et al.  The Secret Life of Patches: A Firefox Case Study , 2012, 2012 19th Working Conference on Reverse Engineering.

[4]  John Riedl,et al.  Is seeing believing?: how recommender system interfaces affect users' opinions , 2003, CHI '03.

[5]  Gail C. Murphy,et al.  Determining Implementation Expertise from Bug Reports , 2007, Fourth International Workshop on Mining Software Repositories (MSR'07:ICSE Workshops 2007).

[6]  Timo Koponen,et al.  Life cycle of Defects in Open Source Software Projects , 2006, OSS.

[7]  Laurian M. Chirica,et al.  The entity-relationship model: toward a unified view of data , 1975, SIGF.

[8]  Gang Yin,et al.  Reviewer recommendation for pull-requests in GitHub: What can we learn from code review and bug assignment? , 2016, Inf. Softw. Technol..

[9]  Andy Huber,et al.  Peer reviews in software: a practical guide , 2002, SOEN.

[10]  Andreas Zeller,et al.  How Long Will It Take to Fix This Bug? , 2007, Fourth International Workshop on Mining Software Repositories (MSR'07:ICSE Workshops 2007).

[11]  Karl E. Wiegers,et al.  Peer Reviews in Software: A Practical Guide , 2001 .

[12]  Vipin Balachandran,et al.  Reducing human effort and improving quality in peer code reviews using automatic static analysis and reviewer recommendation , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[13]  J. W. Hunt,et al.  An Algorithm for Differential File Comparison , 2008 .

[14]  Oscar Nierstrasz,et al.  Assigning bug reports using a vocabulary-based expertise model of developers , 2009, 2009 6th IEEE International Working Conference on Mining Software Repositories.

[15]  Volker Gruhn,et al.  Implementation of a Wiki Development Environment , 2016, SoMeT.

[16]  Stéphane Ducasse,et al.  How developers drive software evolution , 2005, Eighth International Workshop on Principles of Software Evolution (IWPSE'05).

[17]  Michael Gertz,et al.  Expertise identification and visualization from CVS , 2008, MSR '08.

[18]  Zarinah Mohd Kasirun,et al.  Why so complicated? Simple term filtering and weighting for location-based bug report assignment recommendation , 2013, 2013 10th Working Conference on Mining Software Repositories (MSR).

[19]  Burt L. Monroe,et al.  Partial Justification of the Borda Count , 1998 .

[20]  Mik Kersten,et al.  Using task context to improve programmer productivity , 2006, SIGSOFT '06/FSE-14.

[21]  Jesús M. González-Barahona,et al.  Evolution of the core team of developers in libre software projects , 2009, 2009 6th IEEE International Working Conference on Mining Software Repositories.

[22]  Tien N. Nguyen,et al.  Inferring developer expertise through defect analysis , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[23]  Mark S. Ackerman,et al.  Expertise recommender: a flexible recommendation system and architecture , 2000, CSCW '00.

[24]  Ahmed Tamrawi,et al.  Fuzzy set and cache-based approach for bug triaging , 2011, ESEC/FSE '11.

[25]  Shahedul Huq Khandkar,et al.  The role of patch review in software evolution: an analysis of the mozilla firefox , 2009, IWPSE-Evol '09.

[26]  Thomas D. LaToza,et al.  Maintaining mental models: a study of developer work habits , 2006, ICSE.

[27]  Hajimu Iida,et al.  Improving code review effectiveness through reviewer recommendations , 2014, CHASE.

[28]  Gerardo Canfora,et al.  Supporting change request assignment in open source development , 2006, SAC.

[29]  Gail C. Murphy,et al.  Who should fix this bug? , 2006, ICSE.

[30]  Gail C. Murphy,et al.  Reducing the effort of bug report triage: Recommenders for development-oriented decisions , 2011, TSEM.

[31]  Christian Bird,et al.  Automatically Recommending Peer Reviewers in Modern Code Review , 2016, IEEE Transactions on Software Engineering.

[32]  Audris Mockus,et al.  What make long term contributors: Willingness and opportunity in OSS community , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[33]  Hajimu Iida,et al.  Who does what during a code review? Datasets of OSS peer review repositories , 2013, 2013 10th Working Conference on Mining Software Repositories (MSR).

[34]  Marco Aurélio Gerosa,et al.  HISTORICAL ANALYSIS OF MESSAGE CONTENTS TO RECOMMEND ISSUES TO OPEN SOURCE SOFTWARE CONTRIBUTORS , 2014 .

[35]  Dongmei Zhang,et al.  How do software engineers understand code changes?: an exploratory study in industry , 2012, SIGSOFT FSE.

[36]  Thomas Zimmermann,et al.  Improving Code Review by Predicting Reviewers and Acceptance of Patches , 2009 .

[37]  Emily Hill,et al.  Degree-of-knowledge , 2014, ACM Trans. Softw. Eng. Methodol..

[38]  Hajimu Iida,et al.  Who should review my code? A file location-based code-reviewer recommendation approach for Modern Code Review , 2015, 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER).

[39]  James D. Herbsleb,et al.  Social coding in GitHub: transparency and collaboration in an open software repository , 2012, CSCW.

[40]  Emerson R. Murphy-Hill,et al.  A degree-of-knowledge model to capture source code familiarity , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[41]  Audris Mockus,et al.  Expertise Browser: a quantitative approach to identifying expertise , 2002, Proceedings of the 24th International Conference on Software Engineering. ICSE 2002.

[42]  Thomas Zimmermann,et al.  Mining usage expertise from version archives , 2008, MSR '08.

[43]  Jia-Huan He,et al.  CoreDevRec: Automatic Core Member Recommendation for Contribution Evaluation , 2015, Journal of Computer Science and Technology.

[44]  Jaco Geldenhuys,et al.  Finding the Core Developers , 2010, 2010 36th EUROMICRO Conference on Software Engineering and Advanced Applications.

[45]  Q. Mcnemar.  Note on the sampling error of the difference between correlated proportions or percentages , 1947, Psychometrika.

[46]  Gail C. Murphy,et al.  Recommending Emergent Teams , 2007, Fourth International Workshop on Mining Software Repositories (MSR'07:ICSE Workshops 2007).

[47]  Kevin Crowston,et al.  Bug Fixing Practices within Free/Libre Open Source Software Development Teams , 2008, J. Database Manag..