Modern code reviews - Preliminary results of a systematic mapping study

Reviewing source code is a common practice in a modern and collaborative coding environment. In the past few years, the research on modern code reviews has gained interest among practitioners and researchers. The objective of our investigation is to observe the evolution of research related to modern code reviews, identify research gaps and serve as a basis for future research. We use a systematic mapping approach to identify and classify 177 research papers. As preliminary result of our investigation, we present in this paper a classification scheme of the main contributions of modern code review research between 2005 and 2018.

[1]  Foutse Khomh,et al.  Do code review practices impact design quality? A case study of the Qt, VTK, and ITK projects , 2015, 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER).

[2]  Jeffrey C. Carver,et al.  Identifying the characteristics of vulnerable code changes: an empirical study , 2014, SIGSOFT FSE.

[3]  Ayushi Rastogi,et al.  Do Biases Related to Geographical Location Influence Work-Related Decisions in GitHub? , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C).

[4]  Humberto Torres Marques-Neto,et al.  Analyzing The Impact Of Feedback In GitHub On The Software Developer's Mood , 2018, SEKE.

[5]  Ashish Sureka,et al.  Mining Peer Code Review System for Computing Effort and Contribution Metrics for Patch Reviewers , 2014, 2014 IEEE 4th Workshop on Mining Unstructured Data.

[6]  Kathryn T. Stolee,et al.  Evaluating how static analysis tools can reduce code review effort , 2017, 2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[7]  Kai Petersen,et al.  Guidelines for conducting systematic mapping studies in software engineering: An update , 2015, Inf. Softw. Technol..

[8]  Gang Yin,et al.  An Empirical Study of Reviewer Recommendation in Pull-based Development Model , 2017, Internetware.

[9]  Armin Heinzl,et al.  Peer-Based Quality Assurance in Information Systems Development: A Transactive Memory Perspective , 2013, ICIS.

[10]  Shinichi Oeda,et al.  Development of a Check Sheet for Code-review towards Improvement of Skill Level of Novice Programmers , 2018, KES.

[11]  Gaurav Pahwa,et al.  Code review analysis of software system using machine learning techniques , 2017, 2017 11th International Conference on Intelligent Systems and Control (ISCO).

[12]  Krzysztof Stencel,et al.  Profile based recommendation of code reviewers , 2018, Journal of Intelligent Information Systems.

[13]  Chanchal Kumar Roy,et al.  CORRECT: Code Reviewer Recommendation in GitHub Based on Cross-Project and Technology Experience , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C).

[14]  Thomas Grechenig,et al.  On the understanding of programs with continuous code reviews , 2013, 2013 21st International Conference on Program Comprehension (ICPC).

[15]  Yuki Ueda,et al.  How is IF Statement Fixed Through Code Review? A Case Study of Qt Project , 2017, 2017 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW).

[16]  Armin Heinzl,et al.  The Impact of Peer-Based Software Reviews on Team Performance: The Role of Feedback and Transactive Memory Systems , 2012, ICIS.

[17]  Gail C. Murphy,et al.  The Structure of Software Design Discussions , 2018, 2018 IEEE/ACM 11th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE).

[18]  Anthony F. Norcio,et al.  The Effect of Human Memory Organization on Code Reviews under Different Single and Pair Code Reviewing Scenarios , 2005 .

[19]  Fabio Palomba,et al.  Information Needs in Contemporary Code Review , 2018, Proc. ACM Hum. Comput. Interact..

[20]  Gang Yin,et al.  RevRec: A two-layer reviewer recommendation algorithm in pull-based development model , 2018 .

[21]  Jeffrey C. Carver,et al.  Impact of developer reputation on code review outcomes in OSS projects: an empirical investigation , 2014, ESEM '14.

[22]  Yuming Zhou,et al.  The impact of continuous integration on other software development practices: A large-scale empirical study , 2017, 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[23]  Stephen H. Edwards,et al.  Misunderstandings about object-oriented design: experiences using code reviews , 2008, SIGCSE '08.

[24]  Irit Hadar,et al.  Gamifying Software Engineering Tasks Based on Cognitive Principles: The Case of Code Review , 2015, 2015 IEEE/ACM 8th International Workshop on Cooperative and Human Aspects of Software Engineering.

[25]  Christian Bird,et al.  Lessons Learned from Building and Deploying a Code Review Analytics Platform , 2015, 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories.

[26]  Christian Bird,et al.  CodeFlow: Improving the Code Review Process at Microsoft , 2018, ACM Queue.

[27]  Miryung Kim,et al.  RefDistiller: a refactoring aware code review tool for inspecting manual refactoring edits , 2014, SIGSOFT FSE.

[28]  Masateru Tsunoda,et al.  WAP: Does Reviewer Age Affect Code Review Performance? , 2017, 2017 IEEE 28th International Symposium on Software Reliability Engineering (ISSRE).

[29]  Ken-ichi Matsumoto,et al.  Pilot study of collective decision-making in the code review process , 2015, CASCON.

[30]  Kurt Schneider,et al.  Comparing pre‐commit reviews and post‐commit reviews using process simulation , 2017, J. Softw. Evol. Process..

[31]  Jia-Huan He,et al.  CoreDevRec: Automatic Core Member Recommendation for Contribution Evaluation , 2015, Journal of Computer Science and Technology.

[32]  Tobias Dürschmid,et al.  Continuous Code Reviews: A Social Coding tool for Code Reviews inside the IDE , 2017, Programming.

[33]  Gang Yin,et al.  Reviewer Recommender of Pull-Requests in GitHub , 2014, 2014 IEEE International Conference on Software Maintenance and Evolution.

[34]  Yasutaka Kamei,et al.  Effect of review and patch development experience in the chromium project's patch review time , 2015 .

[35]  Harald C. Gall,et al.  An approach for collaborative code reviews using multi-touch technology , 2012, 2012 5th International Workshop on Co-operative and Human Aspects of Software Engineering (CHASE).

[36]  Ig Ibert Bittencourt,et al.  Does peer assessment in on-line learning environments work? A systematic review of the literature , 2016, Computers in Human Behavior.

[37]  Gina Venolia,et al.  Can peer code reviews be exploited for later information needs? , 2009, 2009 31st International Conference on Software Engineering - Companion Volume.

[38]  Vipin Balachandran,et al.  Reducing human effort and improving quality in peer code reviews using automatic static analysis and reviewer recommendation , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[39]  Nicole Novielli,et al.  Confusion Detection in Code Reviews , 2017, 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME).

[40]  Kenichi Matsumoto,et al.  The impact of human factors on the participation decision of reviewers in modern code review , 2018, Empirical Software Engineering.

[41]  Michele Lanza,et al.  ViDI: The Visual Design Inspector , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[42]  Roel Wieringa,et al.  Requirements engineering paper classification and evaluation criteria: a proposal and a discussion , 2005, Requirements Engineering.

[43]  Hajimu Iida,et al.  Who should review my code? A file location-based code-reviewer recommendation approach for Modern Code Review , 2015, 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER).

[44]  Jacek Dajda,et al.  Experimental Validation of Source Code Reviews on Mobile Devices , 2017, ICCSA.

[45]  Yuki Ueda,et al.  How are IF-Conditional Statements Fixed Through Peer CodeReview? , 2018, IEICE Trans. Inf. Syst..

[46]  Sumaira Nazir,et al.  Challenges and Benefits of Modern Code Review-Systematic Literature Review Protocol , 2018, 2018 International Conference on Smart Computing and Electronic Enterprise (ICSCEE).

[47]  Naoyasu Ubayashi,et al.  A Study of the Quality-Impacting Practices of Modern Code Review at Sony Mobile , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C).

[48]  Volker Gruhn,et al.  Automatically recommending code reviewers based on their expertise: An empirical comparison , 2016, 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE).

[49]  Kurt Schneider,et al.  On the Optimal Order of Reading Source Code Changes for Review , 2017, 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME).

[50]  Zhi Jin,et al.  MCT: A tool for commenting programs by multimedia comments , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[51]  Carlos José Pereira de Lucena,et al.  Identifying Code Smells with Collaborative Practices: A Controlled Experiment , 2016, 2016 X Brazilian Symposium on Software Components, Architectures and Reuse (SBCARS).

[52]  Michael W. Godfrey,et al.  The influence of non-technical factors on code review , 2013, 2013 20th Working Conference on Reverse Engineering (WCRE).

[53]  Eitan Farchi,et al.  SeeCode - A Code Review Plug-in for Eclipse , 2009, Haifa Verification Conference.

[54]  Darja Smite,et al.  Software Architects in Large-Scale Distributed Projects: An Ericsson Case Study , 2016, IEEE Software.

[55]  David Lo,et al.  Who should review this change?: Putting text and file location analyses together for more accurate recommendations , 2015, 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME).

[56]  Michael W. Godfrey,et al.  Investigating code review quality: Do people and participation matter? , 2015, 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME).

[57]  Arie van Deursen,et al.  Visualizing code and coverage changes for code review , 2016, SIGSOFT FSE.

[58]  Aziz Nanthaamornphong,et al.  Empirical evaluation of code smells in open source projects: preliminary results , 2016, IWoR@ASE.

[59]  Kenichi Matsumoto,et al.  Do Review Feedbacks Influence to a Contributor's Time Spent on OSS Projects? , 2018, 2018 IEEE International Conference on Big Data, Cloud Computing, Data Science & Engineering (BCD).

[60]  Benjamin Leiding,et al.  Ensuring Resource Trust and Integrity in Web Browsers Using Blockchain Technology , 2018, CAiSE Workshops.

[61]  Sunghun Kim,et al.  Partitioning Composite Code Changes to Facilitate Code Review , 2015, 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories.

[62]  Gang Yin,et al.  Who Should Review this Pull-Request: Reviewer Recommendation to Expedite Crowd Collaboration , 2014, 2014 21st Asia-Pacific Software Engineering Conference.

[63]  Yuriy Tymchuk,et al.  Treating software quality as a first-class entity , 2015, 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME).

[64]  Anderson Belgamo,et al.  An Overview of Experimental Studies on Software Inspection Process , 2013, ICEIS.

[65]  Shane McIntosh,et al.  An empirical study of the impact of modern code review practices on software quality , 2015, Empirical Software Engineering.

[66]  Scott D. Fleming,et al.  CFar: A Tool to Increase Communication, Productivity, and Review Quality in Collaborative Code Reviews , 2018, CHI.

[67]  Alberto Bacchelli,et al.  Expectations, outcomes, and challenges of modern code review , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[68]  Hajimu Iida,et al.  ReDA: A Web-Based Visualization Tool for Analyzing Modern Code Review Dataset , 2014, 2014 IEEE International Conference on Software Maintenance and Evolution.

[69]  Jia-Huan He,et al.  Who should comment on this pull request? Analyzing attributes for more accurate commenter recommendation in pull-based development , 2017, Inf. Softw. Technol..

[70]  Gregorio Robles,et al.  Software Development Analytics for Xen: Why and How , 2019, IEEE Software.

[71]  Christian Bird,et al.  Automatically Recommending Peer Reviewers in Modern Code Review , 2016, IEEE Transactions on Software Engineering.

[72]  Hajimu Iida,et al.  Investigating Code Review Practices in Defective Files: An Empirical Study of the Qt System , 2015, 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories.

[73]  Liang Chen,et al.  EARec: Leveraging Expertise and Authority for Pull-Request Reviewer Recommendation in GitHub , 2016, 2016 IEEE/ACM 3rd International Workshop on CrowdSourcing in Software Engineering (CSI-SE).

[74]  Gerard J. Holzmann,et al.  SCRUB: a tool for code reviews , 2010, Innovations in Systems and Software Engineering.

[75]  Nava Tintarev,et al.  Does Reviewer Recommendation Help Developers , 2020 .

[76]  Yuki Ueda,et al.  The Impact of a Low Level of Agreement Among Reviewers in a Code Review Process , 2016, OSS.

[77]  Jesús M. González-Barahona,et al.  Analyzing Gerrit Code Review Parameters with Bicho , 2014, Electron. Commun. Eur. Assoc. Softw. Sci. Technol..

[78]  Giuliano Antoniol,et al.  Would static analysis tools help developers with code reviews? , 2015, 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER).

[79]  Huzefa H. Kagdi,et al.  Feedback Topics in Modern Code Review: Automatic Identification and Impact on Changes , 2018, SEKE.

[80]  Meng Xia,et al.  Exploring how software developers work with mention bot in GitHub , 2018, CCF Transactions on Pervasive Computing and Interaction.

[81]  Henrique Henriques,et al.  Code review tool for Visual Programming Languages , 2018, 2018 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[82]  Xiaoping Fan,et al.  Topic-Based Integrator Matching for Pull Request , 2017, GLOBECOM 2017 - 2017 IEEE Global Communications Conference.

[83]  Hajimu Iida,et al.  Improving code review effectiveness through reviewer recommendations , 2014, CHASE.

[84]  Leonardo Gresta Paulino Murta,et al.  Developers assignment for analyzing pull requests , 2015, SAC.

[85]  Eunjoo Lee,et al.  Understanding Review Expertise of Developers: A Reviewer Recommendation Approach Based on Latent Dirichlet Allocation , 2018, Symmetry.

[86]  Miryung Kim,et al.  Critics: an interactive code review tool for searching and inspecting systematic changes , 2014, FSE 2014.

[87]  Gail C. Murphy,et al.  Removing stagnation from modern code review , 2016, SPLASH.

[88]  Hajimu Iida,et al.  Revisiting Code Ownership and Its Relationship with Software Quality in the Scope of Modern Code Review , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[89]  Kurt Schneider,et al.  On the Need for a New Generation of Code Review Tools , 2016, PROFES.

[90]  Gabriele Bavota,et al.  Four eyes are better than two: On the impact of code reviews on software quality , 2015, 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME).

[91]  Andrew Meneely,et al.  An empirical investigation of socio-technical code review metrics and security vulnerabilities , 2014, SSE@SIGSOFT FSE.

[92]  Felix Raab,et al.  Collaborative code reviews on interactive surfaces , 2011, ECCE.

[93]  Jesús M. González-Barahona,et al.  Code Review Analytics: WebKit as Case Study , 2014, OSS.

[94]  Emerson R. Murphy-Hill,et al.  Towards refactoring-aware code review , 2014, CHASE.

[95]  Jesús M. González-Barahona,et al.  Using Metrics to Track Code Review Performance , 2017, EASE.

[96]  Katsuro Inoue,et al.  Visualization of Inter-Module Dataflow through Global Variables for Source Code Review , 2018, IEICE Trans. Inf. Syst..

[97]  Leonardo Gresta Paulino Murta,et al.  What factors influence the reviewer assignment to pull requests? , 2018, Inf. Softw. Technol..

[98]  Xuesong Zhang,et al.  Design and Implementation of Java Sniper: A Community-Based Software Code Review Web Solution , 2011, 2011 44th Hawaii International Conference on System Sciences.

[99]  Katsuro Inoue,et al.  Search-Based Peer Reviewers Recommendation in Modern Code Review , 2016, 2016 IEEE International Conference on Software Maintenance and Evolution (ICSME).

[100]  Michael W. Godfrey,et al.  Investigating technical and non-technical factors influencing modern code review , 2015, Empirical Software Engineering.

[101]  Irit Hadar,et al.  Let's Make it Fun: Gamifying and Formalizing Code Review , 2016, ENASE.