A Longitudinal Analysis of Bug Handling Across Eclipse Releases

Large open source software projects, like Eclipse, follow a continuous software development process, with a regular release cycle. During each release, new bugs are reported, triaged and resolved. Previous studies have focused on various aspects of bug fixing, such as bug triaging, bug prediction, and bug process analysis. Most studies, however, do not distinguish between what happens before and after each scheduled release. We are also unaware of studies that compare bug fixing activities across different project releases. This paper presents an empirical analysis of the bug handling process of Eclipse over a 15-year period, considering 138K bug reports from Bugzilla, including 16 annual Eclipse releases and two quarterly releases in 2018. We compare the bug resolution rate, the fixing rate, the bug triaging time and the fixing time before and after each release date, and we study the possible impact of "release pressure". Among others, our results reveal that Eclipse bug handling activity is improving over time, with an important decrease in the number of reported bugs before releases, an increase in the bug fixing rate and an increasingly balanced bug handling workload before and after releases. The recent transition from an annual to a quarterly release cycle continued to improve the bug handling process.

[1]  Liudmila Ulanova,et al.  An Empirical Analysis of Bug Reports and Bug Fixing in Open Source Android Apps , 2013, 2013 17th European Conference on Software Maintenance and Reengineering.

[2]  Jun Yan,et al.  Automatic Bug Triage using Semi-Supervised Text Classification , 2017, SEKE.

[3]  Harald C. Gall,et al.  Predicting the fix time of bugs , 2010, RSSE '10.

[4]  Liang Gong,et al.  Predicting bug-fixing time: An empirical study of commercial software projects , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[5]  Douglas G Altman,et al.  The logrank test , 2004, BMJ : British Medical Journal.

[6]  Amit Kumar,et al.  Evolution of developer social network and its impact on bug fixing process , 2013, ISEC.

[7]  Jeffrey D. Kromrey,et al.  Robust Confidence Intervals for Effect Sizes: A Comparative Study of Cohen's d and Cliff's Delta Under Non-normality and Heterogeneous Variances , 2004 .

[8]  Bart Goethals,et al.  Predicting the severity of a reported bug , 2010, 2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010).

[9]  Westley Weimer,et al.  Modeling bug report quality , 2007, ASE '07.

[10]  Foutse Khomh,et al.  Do faster releases improve software quality? An empirical case study of Mozilla Firefox , 2012, 2012 9th IEEE Working Conference on Mining Software Repositories (MSR).

[11]  Ioannis Stamelos,et al.  Evaluating the Quality of Open Source Software , 2009, SQM@CSMR.

[12]  Sarfraz Khurshid,et al.  Understanding the triaging and fixing processes of long lived bugs , 2015, Inf. Softw. Technol..

[13]  Zarinah Mohd Kasirun,et al.  Why so complicated? Simple term filtering and weighting for location-based bug report assignment recommendation , 2013, 2013 10th Working Conference on Mining Software Repositories (MSR).

[14]  David Lo,et al.  Accurate developer recommendation for bug resolution , 2013, 2013 20th Working Conference on Reverse Engineering (WCRE).

[15]  Christoph Treude,et al.  The impact of rapid release cycles on the integration delay of fixed issues , 2018, Empirical Software Engineering.

[16]  He Jiang,et al.  Towards Effective Bug Triage with Software Data Reduction Techniques , 2017, IEEE Transactions on Knowledge and Data Engineering.

[17]  Andreas Zeller,et al.  How Long Will It Take to Fix This Bug? , 2007, Fourth International Workshop on Mining Software Repositories (MSR'07:ICSE Workshops 2007).

[18]  David Lo,et al.  Dual analysis for recommending developers to resolve bugs , 2015, J. Softw. Evol. Process..

[19]  Ying Zou,et al.  Studying the fix-time for bugs in large open source projects , 2011, Promise '11.

[20]  Gerardo Canfora,et al.  Supporting change request assignment in open source development , 2006, SAC.

[21]  Gail C. Murphy,et al.  Who should fix this bug? , 2006, ICSE.

[22]  John Anvik,et al.  Automating bug report assignment , 2006, ICSE.

[23]  Emerson R. Murphy-Hill,et al.  The design of bug fixes , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[24]  Foutse Khomh,et al.  An Empirical Study on Factors Impacting Bug Fixing Time , 2012, 2012 19th Working Conference on Reverse Engineering.

[25]  Juan Fernández-Ramil,et al.  The evolution of Eclipse , 2008, 2008 IEEE International Conference on Software Maintenance.

[26]  Alexander Serebrenik,et al.  Survival of Eclipse third-party plug-ins , 2012, 2012 28th IEEE International Conference on Software Maintenance (ICSM).

[27]  Monika Gupta,et al.  Improving Software Maintenance Using Process Mining and Predictive Analytics , 2017, 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME).

[28]  Uirá Kulesza,et al.  The Impact of Switching to a Rapid Release Cycle on the Integration Delay of Addressed Issues - An Empirical Study of the Mozilla Firefox Project , 2016, 2016 IEEE/ACM 13th Working Conference on Mining Software Repositories (MSR).

[29]  P. Grambsch.  Survival and Event History Analysis: A Process Point of View by AALEN, O. O., BORGAN, O., and GJESSING, H. K. , 2009 .

[30]  Thomas Zimmermann,et al.  Improving bug triage with bug tossing graphs , 2009, ESEC/FSE '09.

[31]  N. Cliff.  Dominance statistics: Ordinal analyses to answer ordinal questions. , 1993 .

[32]  Sarfraz Khurshid,et al.  Are These Bugs Really "Normal"? , 2015, 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories.

[33]  Lucas D. Panjer.  Predicting Eclipse Bug Lifetimes , 2007, Fourth International Workshop on Mining Software Repositories (MSR'07:ICSE Workshops 2007).

[34]  Andy Zaidman,et al.  Assessment of issue handling efficiency , 2010, 2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010).

[35]  Hao Hu,et al.  Effective Bug Triage Based on Historical Bug-Fix Information , 2014, 2014 IEEE 25th International Symposium on Software Reliability Engineering.

[36]  Weiqiang Zhang,et al.  An Empirical Study of Bug Fixing Rate , 2015, 2015 IEEE 39th Annual Computer Software and Applications Conference.

[37]  Foutse Khomh,et al.  Understanding the impact of rapid releases on software quality , 2015, Empirical Software Engineering.

[38]  Minghui Zhou,et al.  Be careful of when: an empirical study on time-related misuse of issue tracking data , 2018, ESEC/SIGSOFT FSE.