Pull Request Latency Explained: An Empirical Overview

Pull request latency evaluation is an essential application of effort evaluation in the pull-based development scenario. It can help the reviewers sort the pull request queue, remind developers about the review processing time, speed up the review process and accelerate software development. There is a lack of work that systematically organizes the factors that affect pull request latency. Also, there is no related work discussing the differences and variations in characteristics in different scenarios and contexts. In this paper, we collected relevant factors through a literature review approach. Then we assessed their relative importance in five scenarios and six different contexts using the mixed-effects linear regression model. We find that the relative importance of factors differs in different scenarios, e.g., the first response time of the reviewer is most important when there exist comments. Meanwhile, the number of commits in a pull request has a more significant impact on pull request latency when closing than submitting due to changes in contributions brought about by the review process.

[1]  Georgios Gousios,et al.  Work Practices and Challenges in Pull-Based Development: The Integrator's Perspective , 2014, ICSE.

[2]  Michael W. Godfrey,et al.  Studying Pull Request Merges: A Case Study of Shopify's Active Merchant , 2017, 2018 IEEE/ACM 40th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP).

[3]  Georgios Gousios,et al.  Pull Request Decision Explained: An Empirical Overview , 2021, ArXiv.

[4]  Pearl Brereton,et al.  Performing systematic literature reviews in software engineering , 2006, ICSE.

[5]  Ting Wang,et al.  Duplicate Pull Request Detection: When Time Matters , 2019, Internetware.

[6]  Gabriele Bavota,et al.  A Study on the Interplay between Pull Request Review and Continuous Integration Builds , 2019, 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER).

[7]  Christoph Treude,et al.  Automatic Generation of Pull Request Descriptions , 2019, 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[8]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[9]  Jeffrey C. Carver,et al.  Impact of developer reputation on code review outcomes in OSS projects: an empirical investigation , 2014, ESEM '14.

[10]  Luke Church,et al.  Modern Code Review: A Case Study at Google , 2017, 2018 IEEE/ACM 40th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP).

[11]  Emerson R. Murphy-Hill,et al.  Investigating the Effects of Gender Bias on GitHub , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE).

[12]  Yuming Zhou,et al.  The impact of continuous integration on other software development practices: A large-scale empirical study , 2017, 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[13]  Michael W. Godfrey,et al.  The Secret Life of Patches: A Firefox Case Study , 2012, 2012 19th Working Conference on Reverse Engineering.

[14]  Michael W. Godfrey,et al.  Investigating technical and non-technical factors influencing modern code review , 2015, Empirical Software Engineering.

[15]  Jeffrey C. Carver,et al.  Are One-Time Contributors Different? A Comparison to Core and Periphery Developers in FLOSS Repositories , 2017, 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM).

[16]  Gang Yin,et al.  A Dataset of Duplicate Pull-Requests in GitHub , 2018, 2018 IEEE/ACM 15th International Conference on Mining Software Repositories (MSR).

[17]  Kathryn T. Stolee,et al.  Evaluating how static analysis tools can reduce code review effort , 2017, 2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[18]  Jacob Cohen,et al.  Applied multiple regression/correlation analysis for the behavioral sciences , 1979 .

[19]  Li Zhang,et al.  What are the Characteristics of Reopened Pull Requests? A Case Study on Open Source Projects in GitHub , 2019, IEEE Access.

[20]  Arie van Deursen,et al.  An exploratory study of the pull-based software development model , 2014, ICSE.

[21]  Claes Wohlin,et al.  Systematic literature studies: Database searches vs. backward snowballing , 2012, Proceedings of the 2012 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement.

[22]  Øyvind Langsrud,et al.  ANOVA for unbalanced data: Use Type II instead of Type III sums of squares , 2003, Stat. Comput..

[23]  Georgios Gousios,et al.  Automatically Prioritizing Pull Requests , 2015, 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories.

[24]  Chetan Bansal,et al.  Predicting pull request completion time: a case study on large scale cloud services , 2019, ESEC/SIGSOFT FSE.

[25]  Gang Yin,et al.  Who Should Review this Pull-Request: Reviewer Recommendation to Expedite Crowd Collaboration , 2014, 2014 21st Asia-Pacific Software Engineering Conference.

[26]  Ayushi Rastogi,et al.  On the Shoulders of Giants: A New Dataset for Pull-based Development Research , 2020, 2020 IEEE/ACM 17th International Conference on Mining Software Repositories (MSR).

[27]  Gang Yin,et al.  Are You Still Working on This? An Empirical Study on Pull Request Abandonment , 2022, IEEE Transactions on Software Engineering.

[28]  S. Apel,et al.  On the Influence of Developer Coreness on Patch Acceptance: A Survival Analysis , 2020 .

[29]  Daniel M. Germán,et al.  Will my patch make it? And how fast? Case study on the Linux kernel , 2013, 2013 10th Working Conference on Mining Software Repositories (MSR).

[30]  Uirá Kulesza,et al.  Studying the Impact of Adopting Continuous Integration on the Delivery Time of Pull Requests , 2018, 2018 IEEE/ACM 15th International Conference on Mining Software Repositories (MSR).

[31]  Darko Marinov,et al.  Usage, costs, and benefits of continuous integration in open-source projects , 2016, 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE).

[32]  James D. Herbsleb,et al.  Influence of social and technical factors for evaluating contribution in GitHub , 2014, ICSE.

[33]  Gang Yin,et al.  Determinants of pull-based development in the context of continuous integration , 2016, Science China Information Sciences.

[34]  Jia-Huan He,et al.  Who should comment on this pull request? Analyzing attributes for more accurate commenter recommendation in pull-based development , 2017, Inf. Softw. Technol..

[35]  Igor Steinmacher,et al.  Who Gets a Patch Accepted First? Comparing the Contributions of Employees and Volunteers , 2018, 2018 IEEE/ACM 11th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE).

[36]  Shinichi Nakagawa,et al.  A general and simple method for obtaining R2 from generalized linear mixed‐effects models , 2013 .

[37]  Premkumar T. Devanbu,et al.  Wait for It: Determinants of Pull Request Evaluation Latency on GitHub , 2015, 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories.

[38]  Joseph Hilbe,et al.  Data Analysis Using Regression and Multilevel/Hierarchical Models , 2009 .

[39]  Gang Yin,et al.  Bugs and features, do developers treat them differently? , 2018, 2018 International Conference on Artificial Intelligence and Big Data (ICAIBD).