AFID: an automated approach to collecting software faults

We present a new approach for creating repositories of real software faults. We have developed a tool, the Automatic Fault IDentification Tool (AFID), that implements this approach. AFID records both a fault revealing test case and a faulty version of the source code for any crashing faults that the developer discovers and a fault correcting source code change for any crashing faults that the developer corrects. The test cases are a significant contribution, because they enable new research that explores the dynamic behaviors of the software faults. AFID uses an operating system level monitoring mechanism to monitor both the compilation and execution of the application. This technique makes it straightforward for AFID to support a wide range of programming languages and compilers.We present our experience using AFID in a controlled case study and in a real development environment to collect software faults in the internal development of our group’s compiler. The case studies collected several real software faults and validated the basic approach. The longer term internal study revealed weaknesses in using the original version of AFID for real development. This experience led to a number of refinements to the tool for use in real software development. We have collected over 20 real software faults in large programs and continue to collect software faults.

[1]  John Steven,et al.  jRapture: A Capture/Replay tool for observation-based testing , 2000, ISSTA '00.

[2]  Andreas Zeller,et al.  When do changes induce fixes? , 2005, ACM SIGSOFT Softw. Eng. Notes.

[3]  Jong-Deok Choi,et al.  Deterministic replay of Java multithreaded applications , 1998, SPDT '98.

[4]  Daniel C. DuVarney,et al.  Model-carrying code: a practical approach for safe execution of untrusted applications , 2003, SOSP '03.

[5]  Dawson R. Engler,et al.  Execution Generated Test Cases: How to Make Systems Code Crash Itself , 2005, SPIN.

[6]  David Hovemeyer,et al.  Software repository mining with Marmoset , 2005, MSR.

[7]  濱野 純.  入門Git : The fast version control system , 2009 .

[8]  Ben Collins-Sussman,et al.  The subversion project: buiding a better CVS , 2002 .

[9]  Andreas Zeller,et al.  Predicting vulnerable software components , 2007, CCS '07.

[10]  Rahul Vaidya,et al.  AFID: an automated fault identification tool , 2008, ISSTA '08.

[11]  Raphael R. Some,et al.  A software-implemented fault injection methodology for design and validation of system fault tolerance , 2001, 2001 International Conference on Dependable Systems and Networks.

[12]  Andreas Zeller,et al.  Mining metrics to predict component failures , 2006, ICSE.

[13]  Thomas J. LeBlanc,et al.  Debugging Parallel Programs with Instant Replay , 1987, IEEE Transactions on Computers.

[14]  Erez Zadok,et al.  Rapid file system development using ptrace , 2007, ExpCS '07.

[15]  David Hovemeyer,et al.  Software repository mining with Marmoset: an automated programming project snapshot and testing system , 2005, ACM SIGSOFT Softw. Eng. Notes.

[16]  Chadd C. Williams,et al.  Bug Driven Bug Finders , 2004, MSR.

[17]  Thomas Zimmermann,et al.  When do changes induce fixes? On Fridays , 2005 .

[18]  Andreas Zeller,et al.  Yesterday, my program worked. Today, it does not. Why? , 1999, ESEC/FSE-7.

[19]  Thomas Zimmermann,et al.  Extraction of bug localization benchmarks from history , 2007, ASE.

[20]  Michael I. Jordan,et al.  Bug isolation via remote program sampling , 2003, PLDI.

[21]  Benjamin Livshits,et al.  DynaMine: finding common error patterns by mining software revision histories , 2005, ESEC/FSE-13.

[22]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[23]  Yuanyuan Zhou,et al.  BugBench: Benchmarks for Evaluating Bug Detection Tools , 2005 .