Will This Bug-Fixing Change Break Regression Testing?

Context: Software source code is frequently changed for fixing revealed bugs. These bug-fixing changes might introduce unintended system behaviors, which are inconsistent with scenarios of existing regression test cases, and consequently break regression testing. For validating the quality of changes, regression testing is a required process before submitting changes during the development of software projects. Our pilot study shows that 48.7% bug-fixing changes might break regression testing at first run, which means developers have to run regression testing at least a couple of times for 48.7% changes. Such process can be tedious and time consuming. Thus, before running regression test suite, finding these changes and corresponding regression test cases could be helpful for developers to quickly fix these changes and improve the efficiency of regression testing. Goal: This paper proposes bug- fixing change impact prediction (BFCP), for predicting whether a bug-fixing change will break regression testing or not before running regression test cases, by mining software change histories. Method: Our approach employs the machine learning algorithms and static call graph analysis technique. Given a bug-fixing change, BFCP first predicts whether it will break existing regression test cases; second, if the change is predicted to break regression test cases, BFCP can further identify the might-be-broken test cases. Results: Results of experiments on 552 real bug-fixing changes from four large open source projects show that BFCP could achieve prediction precision up to 83.3%, recall up to 92.3%, and F-score up to 81.4%. For identifying the might-be-broken test cases, BFCP could achieve 100% recall.

[1]  Alessandro Orso,et al.  Understanding myths and realities of test-suite evolution , 2012, SIGSOFT FSE.

[2]  David Grove,et al.  A framework for call graph construction algorithms , 2001, TOPL.

[3]  Barbara G. Ryder,et al.  Points-to analysis for Java using annotated constraints , 2001, OOPSLA '01.

[4]  Marco Torchiano,et al.  Impact analysis by means of unstructured knowledge in the context of bug repositories , 2010, ESEM '10.

[5]  Frank Tip,et al.  Finding failure-inducing changes in java programs using change classification , 2006, SIGSOFT '06/FSE-14.

[6]  Song Wang,et al.  DevNet: Exploring Developer Collaboration in Heterogeneous Networks of Bug Repositories , 2013, 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement.

[7]  Richard C. Holt,et al.  Exploring software evolution using spectrographs , 2004, 11th Working Conference on Reverse Engineering.

[8]  Yuriy Brun,et al.  Understanding regression failures through test-passing and test-failing code changes , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[9]  Barbara G. Ryder,et al.  Points-to Analysis for Java Using Annotated Inclusion Constraints , 2000 .

[10]  Dongmei Zhang,et al.  How do software engineers understand code changes?: an exploratory study in industry , 2012, SIGSOFT FSE.

[11]  Yu Luo,et al.  Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-Intensive Systems , 2014, OSDI.

[12]  Wei Le,et al.  Patch verification via multiversion interprocedural control flow graphs , 2014, ICSE.

[13]  Tian Jiang,et al.  Personalized defect prediction , 2013, 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[14]  T. H. Tse,et al.  Adaptive Random Test Case Prioritization , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[15]  Jeffrey Xu Yu,et al.  Matching dependence-related queries in the system dependence graph , 2010, ASE.

[16]  Gregg Rothermel,et al.  An empirical comparison of dynamic impact analysis algorithms , 2004, Proceedings. 26th International Conference on Software Engineering.

[17]  Sashank Dara,et al.  Online Defect Prediction for Imbalanced Data , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[18]  Frank Tip,et al.  Safe-commit analysis to facilitate team software development , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[19]  Shane McIntosh,et al.  Mining Co-change Information to Understand When Build Changes Are Necessary , 2014, 2014 IEEE International Conference on Software Maintenance and Evolution.

[20]  Ayse Basar Bener,et al.  Defect prediction from static code features: current results, limitations, new approaches , 2010, Automated Software Engineering.

[21]  Jeffrey C. Carver,et al.  Impact of Peer Code Review on Peer Impression Formation: A Survey , 2013, 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement.

[22]  Mithun Acharya,et al.  Practical change impact analysis based on static program slicing for industrial software systems , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[23]  Andreas Zeller,et al.  Mining version archives for co-changed lines , 2006, MSR '06.

[24]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[25]  Franz Wotawa,et al.  Impact analysis of SCRs using single and multi-label machine learning classification , 2010, ESEM '10.

[26]  Andy Zaidman,et al.  Aiding Software Developers to Maintain Developer Tests , 2012, 2012 16th European Conference on Software Maintenance and Reengineering.

[27]  Yi Zhang,et al.  Classifying Software Changes: Clean or Buggy? , 2008, IEEE Transactions on Software Engineering.

[28]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[29]  Andreas Zeller,et al.  It's not a bug, it's a feature: How misclassification impacts bug prediction , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[30]  Rongxin Wu,et al.  ReLink: recovering links between bugs and changes , 2011, ESEC/FSE '11.

[31]  Frank Tip,et al.  Change impact analysis for object-oriented programs , 2001, PASTE '01.

[32]  Arie van Deursen,et al.  Mining Software Repositories to Study Co-Evolution of Production & Test Code , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[33]  Shan Lu,et al.  Validating Library Usage Interactively , 2013, CAV.

[34]  Frank Tip,et al.  JUnitMX - A change-aware unit testing tool , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[35]  David Lo,et al.  Identifying Linux bug fixing patches , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[36]  Sungwon Kang,et al.  The Impact of View Histories on Edit Recommendations , 2015, IEEE Transactions on Software Engineering.

[37]  Gregg Rothermel,et al.  Analyzing Regression Test Selection Techniques , 1996, IEEE Trans. Software Eng..

[38]  Andy Podgurski,et al.  JavaPDG: A New Platform for Program Dependence Analysis , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[39]  Michael W. Godfrey,et al.  Evolution in open source software: a case study , 2000, Proceedings 2000 International Conference on Software Maintenance.

[40]  Harald C. Gall,et al.  Do Code and Comments Co-Evolve? On the Relation between Source Code and Comment Changes , 2007, 14th Working Conference on Reverse Engineering (WCRE 2007).

[41]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[42]  Frank Tip,et al.  Chianti: a tool for change impact analysis of java programs , 2004, OOPSLA.