Localizing failure-inducing program edits based on spectrum information

Keeping evolving systems fault free is hard. Change impact analysis is a well-studied methodology for finding faults in evolving systems. For example, in order to help developers identify failure-inducing edits, Chianti extracts program edits as atomic changes between different program versions, selects affected tests, and determines a subset of those changes that might induce test failures. However, identifying real regression faults is challenging for developers since the number of affecting changes related to each test failure may still be too large for manual inspection. This paper presents a novel approach FAULTTRACER which ranks program edits in order to reduce developers' effort in manually inspecting all affecting changes. FAULTTRACER adapts spectrum-based fault localization techniques and applies them in tandem with an enhanced change impact analysis that uses Extended Call Graphs to identify failure-inducing edits more precisely. We evaluate FAULTTRACER using 23 versions of 4 real-world Java programs from the Software Infrastructure Repository. The experimental results show that FAULTTRACER outperforms Chianti in selecting affected tests (slightly better, but handles safety problems of Chianti) as well as in determining affecting changes (with an improvement of approximately 20%). By ranking the affecting changes using spectrum-based test behavior profile, for 14 out of 22 studied failures, FAULTTRACER places a real regression fault within top 3 atomic changes, significantly reducing developers' effort in inspecting potential failure-inducing edits.

[1]  Wei Zhao,et al.  A similarity-aware approach to testing based fault localization , 2005, ASE '05.

[2]  Frank Tip,et al.  Chianti: a tool for change impact analysis of java programs , 2004, OOPSLA.

[3]  Tsong Yueh Chen,et al.  How Well Do Test Case Prioritization Techniques Support Statistical Fault Localization , 2009, 2009 33rd Annual IEEE International Computer Software and Applications Conference.

[4]  Mary Jean Harrold,et al.  An empirical study of the effects of test-suite reduction on fault localization , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[5]  Barbara G. Ryder,et al.  Identifying Failure Causes in Java Programs: An Application of Change Impact Analysis , 2006, IEEE Transactions on Software Engineering.

[6]  Barbara G. Ryder,et al.  Heuristic ranking of java program edits for fault localization , 2007, ISSTA '07.

[7]  Gregg Rothermel,et al.  Whole program path-based dynamic impact analysis , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[8]  Frank Tip,et al.  Change impact analysis for object-oriented programs , 2001, PASTE '01.

[9]  Michael I. Jordan,et al.  Scalable statistical bug isolation , 2005, PLDI '05.

[10]  Andreas Zeller,et al.  Yesterday, my program worked. Today, it does not. Why? , 1999, ESEC/FSE-7.

[11]  Barbara G. Ryder,et al.  Crisp: a debugging tool for Java programs , 2005, 21st IEEE International Conference on Software Maintenance (ICSM'05).

[12]  Alessandro Orso,et al.  Leveraging field data for impact analysis and regression testing , 2003, ESEC/FSE-11.

[13]  Frank Tip,et al.  Finding failure-inducing changes in java programs using change classification , 2006, SIGSOFT '06/FSE-14.

[14]  A.J.C. van Gemund,et al.  On the Accuracy of Spectrum-based Fault Localization , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[15]  Yves Le Traon,et al.  Improving test suites for efficient fault localization , 2006, ICSE.

[16]  Miryung Kim,et al.  Template-based reconstruction of complex refactorings , 2010, 2010 IEEE International Conference on Software Maintenance.

[17]  Gregg Rothermel,et al.  An empirical comparison of dynamic impact analysis algorithms , 2004, Proceedings. 26th International Conference on Software Engineering.

[18]  Robert S. Arnold,et al.  Software Change Impact Analysis , 1996 .

[19]  Raúl A. Santelices,et al.  Lightweight fault-localization using multiple coverage types , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[20]  Tao Xie,et al.  Interactive Fault Localization Using Test Information , 2009, Journal of Computer Science and Technology.

[21]  John T. Stasko,et al.  Visualization of test information to assist fault localization , 2002, ICSE '02.

[22]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.