Automated testing of refactoring engines

Refactorings are behavior-preserving program transformations that improve the design of a program. Refactoring engines are tools that automate the application of refactorings: first the user chooses a refactoring to apply, then the engine checks if the transformation is safe, and if so, transforms the program. Refactoring engines are a key component of modern IDEs, and programmers rely on them to perform refactorings. A bug in the refactoring engine can have severe consequences as it can erroneously change large bodies of source code. We present a technique for automated testing of refactoring engines. Test inputs for refactoring engines are programs. The core of our technique is a framework for iterative generation of structurally complex test inputs. We instantiate the framework to generate abstract syntax trees that represent Java programs. We also create several kinds of oracles to automatically check that the refactoring engine transformed the generated program correctly. We have applied our technique to testing Eclipse and NetBeans, two popular open-source IDEs for Java, and we have exposed 21 new bugs in Eclipse and 24 new bugs in NetBeans.

[1]  Ralf Lämmel,et al.  Controllable Combinatorial Coverage in Grammar-Based Testing , 2006, TestCom.

[2]  Guy L. Steele,et al.  The Java Language Specification , 1996 .

[3]  Brian A. Malloy,et al.  An Interpretation of Purdom's Algorithm forAutomatic Generation of Test Cases , 2001 .

[4]  Sarfraz Khurshid,et al.  TestEra: a novel framework for automated testing of Java programs , 2001, Proceedings 16th Annual International Conference on Automated Software Engineering (ASE 2001).

[5]  Tao Xie,et al.  Augmenting Automatically Generated Unit-Test Suites with Regression Oracle Checking , 2006, ECOOP.

[6]  Darko Marinov,et al.  Automatic testing of software with structurally complex inputs , 2005 .

[7]  Kent L. Beck,et al.  Extreme programming explained - embrace change , 1990 .

[8]  Kent L. Beck.  Smalltalk best practice patterns , 1996 .

[9]  Ippolito Spadafora,et al.  An Automatic Generator for Compiler Testing , 1982, IEEE Transactions on Software Engineering.

[10]  Sarfraz Khurshid,et al.  TestEra: Specification-Based Testing of Java Programs Using SAT , 2004, Automated Software Engineering.

[11]  Koen Claessen,et al.  QuickCheck: a lightweight tool for random testing of Haskell programs , 2000, ICFP.

[12]  Peter M. Maurer,et al.  Generating test data with enhanced context-free grammars , 1990, IEEE Software.

[13]  Sarfraz Khurshid,et al.  Korat: automated testing based on Java predicates , 2002, ISSTA '02.

[14]  A. G. Duncan,et al.  Using attributed grammars to test designs and implementations , 1981, ICSE '81.

[15]  Sarfraz Khurshid,et al.  Generating structurally complex tests from declarative constraints , 2004 .

[16]  William F. Opdyke,et al.  Refactoring object-oriented frameworks , 1992 .

[17]  Matthew B. Dwyer,et al.  Carving differential unit test cases from system test cases , 2006, SIGSOFT '06/FSE-14.

[18]  Emin Gün Sirer,et al.  Using production grammars in software testing , 1999, DSL '99.

[19]  John J. Marciniak,et al.  Encyclopedia of Software Engineering , 1994, Encyclopedia of Software Engineering.

[20]  W. M. McKeeman,et al.  Differential Testing for Software , 1998, Digit. Tech. J..

[21]  Don S. Batory,et al.  The design and implementation of hierarchical software systems with reusable components , 1992, TSEM.

[22]  K. Beck,et al.  Extreme Programming Explained , 2002 .

[23]  Paul Walton Purdom,et al.  A sentence generator for testing parsers , 1972 .

[24]  H. Rice.  Classes of recursively enumerable sets and their decision problems , 1953 .

[25]  Carlo Ghezzi,et al.  Compiler testing using a sentence generator , 1980, Softw. Pract. Exp..

[26]  Sarfraz Khurshid,et al.  Software assurance by bounded exhaustive testing , 2004, IEEE Transactions on Software Engineering.