Selective Bisection Debugging

Bisection debugging, which is based on binary search over software version history, is widely used in practice to identify the bug introducing commit. However, this process can be expensive because it requires costly compilation and test execution at many commits. We introduce a novel technique--selective bisection consisting of test selection and commit selection-- reduce the number of bisection steps, the number of compiler invocations, and the number of executed tests. We evaluated selective bisection on 10 popular open-source projects by performing 25,690 debugging sessions and measuring: 1i¾?savings in number of compiler invocations obtained by commit selection, 2i¾?savings in number of executed tests obtained by test selection, and 3i¾?savings in overall debugging time by selective bisection. Our results show that, in 65% of debugging sessions, commit selection saves between 14% and 71% compiler invocations. Test selection saves 74% of testing effort on average ranging from 42% to 95% compared to when developers do not use any test selection. Finally, we demonstrate that one can save substantial time using selective bisection for large projects.

[1]  Thomas R. Gross.  Bisection Debugging , 1997, AADEBUG.

[2]  Gregg Rothermel,et al.  The Effect of Test Suite Type on Regression Test Selection , 2016, 2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE).

[3]  B. Ness,et al.  Regression containment through source change isolation , 1997, Proceedings Twenty-First Annual International Computer Software and Applications Conference (COMPSAC'97).

[4]  Letha H. Etzkorn,et al.  Bug localization using latent Dirichlet allocation , 2010, Inf. Softw. Technol..

[5]  Peter Zoeteweij,et al.  A practical evaluation of spectrum-based fault localization , 2009, J. Syst. Softw..

[6]  Alessandro Orso,et al.  Regression testing in the presence of non-code changes , 2011, 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation.

[7]  Darko Marinov,et al.  Balancing trade-offs in test-suite reduction , 2014, SIGSOFT FSE.

[8]  Alessandro Orso,et al.  MIMIC: locating and understanding bugs by analyzing mimicked executions , 2014, ASE.

[9]  Hareton K. N. Leung,et al.  Insights into regression testing (software testing) , 1989, Proceedings. Conference on Software Maintenance - 1989.

[10]  Mary Jean Harrold,et al.  Empirical evaluation of the tarantula automatic fault-localization technique , 2005, ASE.

[11]  Alessandro Orso,et al.  Scaling regression testing to large software systems , 2004, SIGSOFT '04/FSE-12.

[12]  Andreas Zeller,et al.  Yesterday, my program worked. Today, it does not. Why? , 1999, ESEC/FSE-7.

[13]  Darko Marinov,et al.  Ekstazi: Lightweight Test Selection , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[14]  Leonardo Mariani,et al.  RADAR: A tool for debugging regression problems in C/C++ Software , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[15]  Gregg Rothermel,et al.  A safe, efficient algorithm for regression test selection , 1993, 1993 Conference on Software Maintenance.

[16]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[17]  Jian Zhou,et al.  Where should the bugs be fixed? More accurate information retrieval-based bug localization based on bug reports , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[18]  Sarfraz Khurshid,et al.  Improving bug localization using structured information retrieval , 2013, 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[19]  Darko Marinov,et al.  Practical regression test selection with dynamic file dependencies , 2015, ISSTA.

[20]  Rupak Majumdar,et al.  Regression Test Selection for Distributed Software Histories , 2014, CAV.

[21]  Gregg Rothermel,et al.  A safe, efficient regression test selection technique , 1997, TSEM.

[22]  Michael I. Jordan,et al.  Scalable statistical bug isolation , 2005, PLDI '05.

[23]  Laurie A. Williams,et al.  An initial study of a lightweight process for change identification and regression test selection when source code is not available , 2005, 16th IEEE International Symposium on Software Reliability Engineering (ISSRE'05).

[24]  Hareton K. N. Leung,et al.  A cost model to compare regression test strategies , 1991, Proceedings. Conference on Software Maintenance 1991.

[25]  Xiangyu Zhang,et al.  Experimental evaluation of using dynamic slices for fault location , 2005, AADEBUG'05.

[26]  Benjamin Livshits,et al.  Automated migration of build scripts using dynamic analysis and search-based refactoring , 2014, OOPSLA.

[27]  Brendan Murphy,et al.  The Art of Testing Less without Sacrificing Quality , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[28]  Gregg Rothermel,et al.  A framework for evaluating regression test selection techniques , 1994, Proceedings of 16th International Conference on Software Engineering.

[29]  Yijun Yu,et al.  Removing false code dependencies to speedup software build processes , 2003, CASCON.

[30]  Sarfraz Khurshid,et al.  Localizing failure-inducing program edits based on spectrum information , 2011, 2011 27th IEEE International Conference on Software Maintenance (ICSM).

[31]  David Lo,et al.  Information retrieval and spectrum based bug localization: better together , 2015, ESEC/SIGSOFT FSE.

[32]  Mark Harman,et al.  Pareto efficient multi-objective test case selection , 2007, ISSTA '07.

[33]  Avinash C. Kak,et al.  Retrieval from software libraries for bug localization: a comparative study of generic and composite text models , 2011, MSR '11.

[34]  Gregg Rothermel,et al.  Analyzing Regression Test Selection Techniques , 1996, IEEE Trans. Software Eng..

[35]  Hung Viet Nguyen,et al.  A topic-based approach for narrowing the search space of buggy files from a bug report , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).