The Power of Bots

Leveraging the pull request model of social coding platforms, Open Source Software (OSS) integrators review developers' contributions, checking aspects like license, code quality, and testability. Some projects use bots to automate predefined, sometimes repetitive tasks, thereby assisting integrators' and contributors' work. Our research investigates the usage and impact of such bots. We sampled 351 popular projects from GitHub and found that 93 (26%) use bots. We classified the bots, collected metrics from before and after bot adoption, and surveyed 228 developers and integrators. Our results indicate that bots perform numerous tasks. Although integrators reported that bots are useful for maintenance tasks, we did not find a consistent, statistically significant difference between before and after bot adoption across the analyzed projects in terms of number of comments, commits, changed files, and time to close pull requests. Our survey respondents deem the current bots as not smart enough and provided insights into the bots' relevance for specific tasks, challenges, and potential new features. We discuss some of the raised suggestions and challenges in light of the literature in order to help GitHub bot designers reuse and test ideas and technologies already investigated in other contexts.

[1]  Bin Xu,et al.  SoBot: facilitating conversation using social media data and a social agent , 2014, CSCW Companion.

[2]  Michihiko Minoh,et al.  Investigation of a Method to Estimate Learners' Interest Level for Agent-Based Conversational e-Learning , 2012, IPMU.

[3]  Ivan Beschastnikh,et al.  Accelerating Software Engineering Research Adoption with Analysis Bots , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering: New Ideas and Emerging Technologies Results Track (ICSE-NIER).

[4]  Alexander Maedche,et al.  Towards Designing Cooperative and Social Conversational Agents for Customer Service , 2017, ICIS.

[5]  Gang Yin,et al.  Reviewer recommendation for pull-requests in GitHub: What can we learn from code review and bug assignment? , 2016, Inf. Softw. Technol..

[6]  Arie van Deursen,et al.  An exploratory study of the pull-based software development model , 2014, ICSE.

[7]  Miguel E. Ruiz,et al.  Usability and effectiveness evaluation of a course-advising chat bot , 2007, ASIST.

[8]  David W. McDonald,et al.  Dissecting a Social Botnet: Growth, Content and Influence in Twitter , 2015, CSCW.

[9]  Marco Aurélio Gerosa,et al.  Almost There: A Study on Quasi-Contributors in Open-Source Software Projects , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[10]  Georgios Gousios,et al.  Work Practices and Challenges in Pull-Based Development: The Integrator's Perspective , 2014, ICSE.

[11]  Luke K. Fryer,et al.  Stimulating and sustaining interest in a language course: An experimental comparison of Chatbot and Human task partners , 2017, Comput. Hum. Behav..

[12]  Margaret-Anne D. Storey,et al.  Disrupting developer productivity one bot at a time , 2016, SIGSOFT FSE.

[13]  Patrick Bii.  Chatbot technology: A possible means of unlocking student potential to learn how to learn , 2013 .

[14]  Marco Aurélio Gerosa,et al.  Social Barriers Faced by Newcomers Placing Their First Contribution in Open Source Software Projects , 2015, CSCW.

[15]  Gail C. Murphy,et al.  Who should fix this bug? , 2006, ICSE.

[16]  Juan de Lara,et al.  The rise of the (modelling) bots: Towards assisted modelling via social networks , 2017, 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[17]  Marco Tulio Valente,et al.  Understanding the Factors That Impact the Popularity of GitHub Repositories , 2016, 2016 IEEE International Conference on Software Maintenance and Evolution (ICSME).

[18]  Ling Xu,et al.  Duplication Detection for Software Bug Reports based on Topic Model , 2016, 2016 9th International Conference on Service Science (ICSS).

[19]  Chris Parnin,et al.  Can automated pull requests encourage software developers to upgrade out-of-date dependencies? , 2017, 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[20]  Premkumar T. Devanbu,et al.  Wait for It: Determinants of Pull Request Evaluation Latency on GitHub , 2015, 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories.

[21]  Volker Gruhn,et al.  Patterns for tearing down contribution barriers to FLOSS projects , 2013, 2013 IEEE 12th International Conference on Intelligent Software Methodologies, Tools and Techniques (SoMeT).

[22]  Susan Bull,et al.  Conversational Agents in E-Learning , 2008, SGAI Conf..

[23]  Juanan Pereira,et al.  Leveraging chatbots to improve self-guided learning through conversational quizzes , 2016, TEEM.

[24]  Catarina Costa,et al.  TIPMerge: recommending experts for integrating changes across branches , 2016, SIGSOFT FSE.

[25]  Vincent Hellendoorn,et al.  Perceived language complexity in GitHub issue discussions and their effect on issue resolution , 2017, 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[26]  André van der Hoek,et al.  A Framework for Understanding Chatbots and Their Future , 2018, 2018 IEEE/ACM 11th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE).

[27]  Ingrid Poncin,et al.  Animated conversational agents and e-consumer productivity: The roles of agents and individual characteristics , 2017, Inf. Manag..

[28]  Christoph Treude,et al.  Overcoming Open Source Project Entry Barriers with a Portal for Newcomers , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[29]  Michele Lanza,et al.  Seahawk: Stack Overflow in the IDE , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[30]  Anita Sarma,et al.  Which bug should I fix: helping new developers onboard a new project , 2011, CHASE '11.

[31]  Eleni Stroulia,et al.  Detecting duplicate bug reports with software engineering domain knowledge , 2015, 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER).

[32]  Thomas Fritz,et al.  Context-Aware Conversational Developer Assistants , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[33]  Emerson R. Murphy-Hill,et al.  Improving developer participation rates in surveys , 2013, 2013 6th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE).

[34]  Marco Aurélio Gerosa,et al.  A systematic literature review on the barriers faced by newcomers to open source software projects , 2015, Inf. Softw. Technol..

[35]  James D. Herbsleb,et al.  Social coding in GitHub: transparency and collaboration in an open software repository , 2012, CSCW.

[36]  Victor Zue,et al.  Conversational interfaces: advances and challenges , 1997, Proceedings of the IEEE.

[37]  Supratip Ghose,et al.  Toward the implementation of a topic specific dialogue based natural language chatbot as an undergraduate advisor , 2013, 2013 International Conference on Informatics, Electronics and Vision (ICIEV).

[38]  N. T. Thomas,et al.  An e-business chatbot using AIML and LSA , 2016, 2016 International Conference on Advances in Computing, Communications and Informatics (ICACCI).

[39]  M. Paffard,et al.  THE TEACHING OF ENGLISH LITERATURE IN SECONDARY SCHOOLS : Part I‐‐A Review of Research since 1945 , 1962 .

[40]  John Riedl,et al.  SuggestBot: using intelligent task routing to help people find work in wikipedia , 2007, IUI '07.

[41]  Chanchal Kumar Roy,et al.  CORRECT: Code Reviewer Recommendation in GitHub Based on Cross-Project and Technology Experience , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C).

[42]  Christoph Treude,et al.  Let Me In: Guidelines for the Successful Onboarding of Newcomers to Open Source Projects , 2019, IEEE Software.

[43]  Yvonne de Kort,et al.  Bots Mind the Social-technical Gap , 2017, ECSCW Exploratory Papers.

[44]  Shari Lawrence Pfleeger,et al.  Preliminary Guidelines for Empirical Research in Software Engineering , 2002, IEEE Trans. Software Eng..

[45]  Leonardo Gresta Paulino Murta,et al.  Automatic assignment of integrators to pull requests: The importance of selecting appropriate attributes , 2018, J. Syst. Softw..

[46]  Stavros N. Demetriadis,et al.  Conversational Agents Improve Peer Learning through Building on Prior Knowledge , 2017, J. Educ. Technol. Soc..

[47]  Margaret-Anne D. Storey,et al.  How Software Developers Mitigate Collaboration Friction with Chatbots , 2017, ArXiv.

[48]  Simon Urli,et al.  How to Design a Program Repair Bot? Insights from the Repairnator Project , 2017, 2018 IEEE/ACM 40th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP).

[49]  Amiram Yehudai,et al.  Example Overflow: Using social media for code recommendation , 2012, 2012 Third International Workshop on Recommendation Systems for Software Engineering (RSSE).

[50]  A. M. Turing,et al.  Computing Machinery and Intelligence , 1950, The Philosophy of Artificial Intelligence.

[51]  Luciana Benotti,et al.  Engaging high school students using chatbots , 2014, ITiCSE '14.

[52]  Diana Pérez-Marín,et al.  Designing and Evaluating Pedagogic Conversational Agents to Teach Children , 2017 .

[53]  Bruce Edmonds,et al.  Oscar: An intelligent conversational agent tutor to estimate learning styles , 2010, International Conference on Fuzzy Systems.

[54]  Marco Aurélio Gerosa,et al.  How Does the Shift to GitHub Impact Project Collaboration? , 2016, 2016 IEEE International Conference on Software Maintenance and Evolution (ICSME).

[55]  Albert A. Angehrn,et al.  Using conversational agents to support the adoption of knowledge sharing practices , 2003, Interact. Comput..

[56]  A. M. Turing,et al.  Computing Machinery and Intelligence , 1950, The Philosophy of Artificial Intelligence.

[57]  James D. Herbsleb,et al.  Let's talk about it: evaluating contributions through discussion in GitHub , 2014, SIGSOFT FSE.

[58]  Georgios Gousios,et al.  Work practices and challenges in pull-based development: the contributor's perspective , 2015, ICSE.

[59]  Christoph Treude,et al.  Augmenting API Documentation with Insights from Stack Overflow , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[60]  Aaron Halfaker,et al.  Operationalizing Conflict and Cooperation between Automated Software Agents in Wikipedia , 2017, Proc. ACM Hum. Comput. Interact..

[61]  Alexander Serebrenik,et al.  Why Developers Are Slacking Off: Understanding How Software Teams Use Slack , 2016, CSCW Companion.

[62]  Marco Aurélio Gerosa,et al.  More Common Than You Think: An In-depth Study of Casual Contributors , 2016, 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER).

[63]  Dirk Heylen,et al.  Open Challenges in Modelling, Analysis and Synthesis of Human Behaviour in Human–Human and Human–Machine Interactions , 2015, Cognitive Computation.

[64]  Gerardo Canfora,et al.  Who is going to mentor newcomers in open source projects? , 2012, SIGSOFT FSE.

[65]  Jonathan Grudin,et al.  Human-computer integration , 2016, Interactions.

[66]  W. Briggs.  Statistical Methods in the Atmospheric Sciences , 2007 .

[67]  Raquel Oliveira Prates,et al.  Here's What I Can Do: Chatbots' Strategies to Convey Their Features to Users , 2017, IHC.

[68]  Shwetak N. Patel,et al.  Convey: Exploring the Use of a Context View for Chatbots , 2018, CHI.

[69]  Karsten Weihe,et al.  Benjamin Franklin’s Decision Method is Acceptable and Helpful with a Conversational Agent , 2015 .

[70]  James D. Herbsleb,et al.  Influence of social and technical factors for evaluating contribution in GitHub , 2014, ICSE.

[71]  Robert Dale,et al.  The return of the chatbots , 2016, Natural Language Engineering.

[72]  Tobias Höllerer,et al.  Botivist: Calling Volunteers to Action using Online Bots , 2015, CSCW.

[73]  Margaret-Anne D. Storey,et al.  Software Bots , 2017, IEEE Software.

[74]  R. Stuart Geiger.  Are computers merely "supporting" cooperative work: towards an ethnography of bot development , 2013, CSCW '13.