Nudge: Accelerating Overdue Pull Requests toward Completion

Pull requests are a key part of the collaborative software development and code review process today. However, pull requests can also slow down the software development process when the reviewer(s) or the author do not actively engage with the pull request. In this work, we design an end-to-end service, Nudge, for accelerating overdue pull requests towards completion by reminding the author or the reviewer(s) to engage with their overdue pull requests. First, we use models based on effort estimation and machine learning to predict the completion time for a given pull request. Second, we use activity detection to reduce false positives. Lastly, we use dependency determination to understand the blocker of the pull request and nudge the appropriate actor(author or reviewer(s)). We also do a correlation analysis to understand the statistical relationship between the pull request completion times and various pull request and developer related attributes. Nudge has been deployed on 147 repositories at Microsoft since 2019. We do a large scale evaluation based on the implicit and explicit feedback we received from sending the Nudge notifications on 8,500 pull requests. We observe significant reduction in completion time, by over 60%, for pull requests which were nudged thus increasing the efficiency of the code review process and accelerating the pull request progression.

[1]  P. Kanakaraja,et al.  Smart home automation using IFTTT and google assistant , 2021 .

[2]  Premkumar T. Devanbu,et al.  Wait for It: Determinants of Pull Request Evaluation Latency on GitHub , 2015, 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories.

[3]  Vipin Balachandran,et al.  Reducing human effort and improving quality in peer code reviews using automatic static analysis and reviewer recommendation , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[4]  Ranjita Bhagwan,et al.  Orca: Differential Bug Localization in Large-Scale Services , 2018, OSDI.

[5]  James D. Herbsleb,et al.  Influence of social and technical factors for evaluating contribution in GitHub , 2014, ICSE.

[6]  Ranjita Bhagwan,et al.  Rex: Preventing Bugs and Misconfiguration in Large Services Using Correlated Change Analysis , 2020, NSDI.

[7]  Steven Ovadia.  Automate the Internet With “If This Then That” (IFTTT) , 2014 .

[8]  Emerson Murphy-Hill,et al.  Gender differences and bias in open source: pull request acceptance of women versus men , 2017, PeerJ Comput. Sci..

[9]  Barry W. Boehm,et al.  Cost models for future software life cycle processes: COCOMO 2.0 , 1995, Ann. Softw. Eng..

[10]  Forrest Shull,et al.  Local versus Global Lessons for Defect Prediction and Effort Estimation , 2013, IEEE Transactions on Software Engineering.

[11]  Chetan Bansal,et al.  DeCaf: Diagnosing and Triaging Performance Issues in Large-Scale Cloud Services , 2020, 2020 IEEE/ACM 42nd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP).

[12]  Lucas Layman,et al.  Mining software effort data: preliminary analysis of visual studio team system data , 2008, MSR '08.

[13]  Harald Steck,et al.  Item popularity and recommendation accuracy , 2011, RecSys '11.

[14]  Audris Mockus,et al.  Detecting and Characterizing Bots that Commit Code , 2020, 2020 IEEE/ACM 17th International Conference on Mining Software Repositories (MSR).

[15]  Andrzej Wasowski,et al.  Identifying Redundancies in Fork-based Development , 2019, 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER).

[16]  Chetan Bansal,et al.  Predicting pull request completion time: a case study on large scale cloud services , 2019, ESEC/SIGSOFT FSE.

[17]  Paulo Borba,et al.  Understanding predictive factors for merge conflicts , 2020, Inf. Softw. Technol..

[18]  Song Wang,et al.  Large-scale intent analysis for identifying large-review-effort code changes , 2021, Inf. Softw. Technol..

[19]  Barry W. Boehm,et al.  Bayesian Analysis of Empirical Software Engineering Cost Models , 1999, IEEE Trans. Software Eng..

[20]  Ranjita Bhagwan,et al.  WhoDo: automating reviewer suggestions at scale , 2019, ESEC/SIGSOFT FSE.

[21]  Margaret-Anne D. Storey,et al.  Defining and Classifying Software Bots: A Faceted Taxonomy , 2019, 2019 IEEE/ACM 1st International Workshop on Bots in Software Engineering (BotSE).

[22]  Chen Lin,et al.  Spiral of Silence in Recommender Systems , 2019, WSDM.

[23]  Iman Attarzadeh,et al.  Proposing an Enhanced Artificial Neural Network Prediction Model to Improve the Accuracy in Software Effort Estimation , 2012, 2012 Fourth International Conference on Computational Intelligence, Communication Systems and Networks.

[24]  James D. Herbsleb,et al.  BOTse: Bots in Software Engineering (Dagstuhl Seminar 19471) , 2019, Dagstuhl Reports.

[25]  Chetan Bansal,et al.  Building Sankie: An AI Platform for DevOps , 2019, 2019 IEEE/ACM 1st International Workshop on Bots in Software Engineering (BotSE).

[26]  Leonardo Gresta Paulino Murta,et al.  Acceptance factors of pull requests in open-source projects , 2015, SAC.

[27]  Ting Wang,et al.  Duplicate Pull Request Detection: When Time Matters , 2019, Internetware.

[28]  Gang Yin,et al.  Reviewer recommendation for pull-requests in GitHub: What can we learn from code review and bug assignment? , 2016, Inf. Softw. Technol..

[29]  Georgios Gousios,et al.  Relationship between geographical location and evaluation of developer contributions in github , 2018, ESEM.

[30]  Song Wang,et al.  Leveraging Change Intents for Characterizing and Identifying Large-Review-Effort Changes , 2019, PROMISE.

[31]  Daniela Giordano,et al.  Internetworked wrist sensing devices for Pervasive and M-Connected Eldercare , 2021, 2021 IEEE 3rd Global Conference on Life Sciences and Technologies (LifeTech).

[32]  Edward Cutrell,et al.  "Yours is better!": participant response bias in HCI , 2012, CHI.

[33]  Christian Bird,et al.  Code Reviewing in the Trenches: Challenges and Best Practices , 2018, IEEE Software.

[34]  Ahmed E. Hassan,et al.  Towards improving statistical modeling of software engineering data: think locally, act globally! , 2015, Empirical Software Engineering.

[35]  Marvin Wyrich,et al.  Towards an Autonomous Bot for Automatic Source Code Refactoring , 2019, 2019 IEEE/ACM 1st International Workshop on Bots in Software Engineering (BotSE).

[36]  Mohammad R. Khosravi,et al.  An IoT-enabled intelligent automobile system for smart cities , 2020, Internet Things.

[37]  Elliot Soloway,et al.  Where the bugs are , 1985, CHI '85.

[38]  Barry W. Boehm,et al.  Software Engineering Economics , 1993, IEEE Transactions on Software Engineering.

[39]  Michael W. Godfrey,et al.  The influence of non-technical factors on code review , 2013, 2013 20th Working Conference on Reverse Engineering (WCRE).

[40]  Georgios Gousios,et al.  Automatically Prioritizing Pull Requests , 2015, 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories.

[41]  Lionel C. Briand,et al.  An assessment and comparison of common software cost estimation modeling techniques , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).