Automated Behavioral Testing of Refactoring Engines

Refactoring is a transformation that preserves the external behavior of a program and improves its internal quality. Usually, compilation errors and behavioral changes are avoided by preconditions determined for each refactoring transformation. However, to formally define these preconditions and transfer them to program checks is a rather complex task. In practice, refactoring engine developers commonly implement refactorings in an ad hoc manner since no guidelines are available for evaluating the correctness of refactoring implementations. As a result, even mainstream refactoring engines contain critical bugs. We present a technique to test Java refactoring engines. It automates test input generation by using a Java program generator that exhaustively generates programs for a given scope of Java declarations. The refactoring under test is applied to each generated program. The technique uses SafeRefactor, a tool for detecting behavioral changes, as an oracle to evaluate the correctness of these transformations. Finally, the technique classifies the failing transformations by the kind of behavioral change or compilation error introduced by them. We have evaluated this technique by testing 29 refactorings in Eclipse JDT, NetBeans, and the JastAdd Refactoring Tools. We analyzed 153,444 transformations, and identified 57 bugs related to compilation errors, and 63 bugs related to behavioral changes.

[1]  Rohit Gheyi,et al.  Making Program Refactoring Safer , 2010, IEEE Software.

[2]  John B. Goodenough,et al.  Toward a theory of test data selection , 1975 .

[3]  W. M. McKeeman,et al.  Differential Testing for Software , 1998, Digit. Tech. J..

[4]  Torbjörn Ekman,et al.  Challenge proposal: verification of refactorings , 2009, PLPV '09.

[5]  Paul A. Strooper,et al.  Grammar‐based test generation with YouGen , 2011, Softw. Pract. Exp..

[6]  Darko Marinov,et al.  Automated testing of refactoring engines , 2007, ESEC-FSE '07.

[7]  Don S. Batory,et al.  Evolving Object-Oriented Designs with Refactorings , 2004, Automated Software Engineering.

[8]  Luciano Baresi,et al.  TestFul: automatic unit-test generation for Java classes , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[9]  Frank Tip,et al.  Efficiently Refactoring Java Applications to Use Generic Libraries , 2005, ECOOP.

[10]  Sarfraz Khurshid,et al.  Software Assurance by Bounded Exhaustive Testing , 2005, IEEE Trans. Software Eng..

[11]  Daniel Jackson,et al.  Alcoa: the Alloy constraint analyzer , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[12]  Hong Zhu,et al.  Software unit test coverage and adequacy , 1997, ACM Comput. Surv..

[13]  Augusto Sampaio,et al.  Algebraic reasoning for object-oriented programming , 2004, Sci. Comput. Program..

[14]  K. V. Hanford,et al.  Automatic Generation of Test Cases , 1970, IBM Syst. J..

[15]  Carlo Ghezzi,et al.  Compiler testing using a sentence generator , 1980, Softw. Pract. Exp..

[16]  Darko Marinov,et al.  Reducing the Costs of Bounded-Exhaustive Testing , 2009, FASE.

[17]  Frank Tip,et al.  Refactoring support for class library migration , 2005, OOPSLA '05.

[18]  Mik Kersten,et al.  How are Java software developers using the Elipse IDE? , 2006, IEEE Software.

[19]  Bogdan Korel,et al.  Automated regression test generation , 1998, ISSTA '98.

[20]  Sarfraz Khurshid,et al.  TestEra: a novel framework for automated testing of Java programs , 2001, Proceedings 16th Annual International Conference on Automated Software Engineering (ASE 2001).

[21]  Tom Mens,et al.  A survey of software refactoring , 2004, IEEE Transactions on Software Engineering.

[22]  Torbjörn Ekman,et al.  Stepping Stones over the Refactoring Rubicon , 2009, ECOOP.

[23]  Michael D. Ernst,et al.  Feedback-Directed Random Test Generation , 2007, 29th International Conference on Software Engineering (ICSE'07).

[24]  Torbjörn Ekman,et al.  Sound and extensible renaming for java , 2008, OOPSLA.

[25]  Augusto Sampaio,et al.  Laws of Object-Orientation with Reference Semantics , 2008, 2008 Sixth IEEE International Conference on Software Engineering and Formal Methods.

[26]  Huiqing Li,et al.  Testing Erlang Refactorings with QuickCheck , 2008, IFL.

[27]  Tom Mens,et al.  Formalising Behaviour Preserving Program Transformations , 2002, ICGT.

[28]  Ippolito Spadafora,et al.  An Automatic Generator for Compiler Testing , 1982, IEEE Transactions on Software Engineering.

[29]  Sarfraz Khurshid,et al.  Test generation through programming in UDITA , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[30]  Frank Tip,et al.  Correct Refactoring of Concurrent Java Code , 2010, ECOOP.

[31]  Jeffrey Overbey,et al.  Differential precondition checking: A lightweight, reusable analysis for refactoring tools , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[32]  Friedrich Steimann,et al.  From Public to Private to Absent: Refactoring Java Programs under Constrained Accessibility , 2009, ECOOP.

[33]  Rohit Gheyi,et al.  Identifying overly strong conditions in refactoring implementations , 2011, 2011 27th IEEE International Conference on Software Maintenance (ICSM).

[34]  Ralph E. Johnson,et al.  The role of refactorings in API evolution , 2005, 21st IEEE International Conference on Software Maintenance (ICSM'05).

[35]  Alessandro Orso,et al.  Automated Behavioral Regression Testing , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[36]  Oege de Moor,et al.  Specifying and implementing refactorings , 2010, OOPSLA.

[37]  Amer Diwan,et al.  Program Metamorphosis , 2009, ECOOP.