A Static Approach to Prioritizing JUnit Test Cases

Test case prioritization is used in regression testing to schedule the execution order of test cases so as to expose faults earlier in testing. Over the past few years, many test case prioritization techniques have been proposed in the literature. Most of these techniques require data on dynamic execution in the form of code coverage information for test cases. However, the collection of dynamic code coverage information on test cases has several associated drawbacks including cost increases and reduction in prioritization precision. In this paper, we propose an approach to prioritizing test cases in the absence of coverage information that operates on Java programs tested under the JUnit framework-an increasingly popular class of systems. Our approach, JUnit test case Prioritization Techniques operating in the Absence of coverage information (JUPTA), analyzes the static call graphs of JUnit test cases and the program under test to estimate the ability of each test case to achieve code coverage, and then schedules the order of these test cases based on those estimates. To evaluate the effectiveness of JUPTA, we conducted an empirical study on 19 versions of four Java programs ranging from 2K-80K lines of code, and compared several variants of JUPTA with three control techniques, and several other existing dynamic coverage-based test case prioritization techniques, assessing the abilities of the techniques to increase the rate of fault detection of test suites. Our results show that the test suites constructed by JUPTA are more effective than those in random and untreated test orders in terms of fault-detection effectiveness. Although the test suites constructed by dynamic coverage-based techniques retain fault-detection effectiveness advantages, the fault-detection effectiveness of the test suites constructed by JUPTA is close to that of the test suites constructed by those techniques, and the fault-detection effectiveness of the test suites constructed by some of JUPTA's variants is better than that of the test suites constructed by several of those techniques.

[1]  Gregg Rothermel,et al.  The impact of software evolution on code coverage information , 2001, Proceedings IEEE International Conference on Software Maintenance. ICSM 2001.

[2]  Paolo Tonella,et al.  Using the Case-Based Ranking Methodology for Test Case Prioritization , 2006, 2006 22nd IEEE International Conference on Software Maintenance.

[3]  Hareton K. N. Leung,et al.  Insights into regression testing (software testing) , 1989, Proceedings. Conference on Software Maintenance - 1989.

[4]  Gregg Rothermel,et al.  Prioritizing JUnit Test Cases: An Empirical Assessment and Cost-Benefits Analysis , 2006, Empirical Software Engineering.

[5]  Amitabh Srivastava,et al.  Effectively prioritizing tests in development environment , 2002, ISSTA '02.

[6]  David Grove,et al.  Optimization of Object-Oriented Programs Using Static Class Hierarchy Analysis , 1995, ECOOP.

[7]  Matthew Arnold,et al.  A framework for reducing the cost of instrumented code , 2001, PLDI '01.

[8]  Gregg Rothermel,et al.  Using sensitivity analysis to create simplified economic models for regression testing , 2008, ISSTA '08.

[9]  Jeffrey K. Hollingsworth,et al.  Efficient instrumentation for code coverage testing , 2002, ISSTA '02.

[10]  Gregg Rothermel,et al.  Incorporating varying test costs and fault severities into test case prioritization , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[11]  A. Jefferson Offutt,et al.  MuJava: an automated class mutation system , 2005, Softw. Test. Verification Reliab..

[12]  Mary Lou Soffa,et al.  TimeAware test suite prioritization , 2006, ISSTA '06.

[13]  Joseph Robert Horgan,et al.  A study of effective regression testing in practice , 1997, Proceedings The Eighth International Symposium on Software Reliability Engineering.

[14]  Gregg Rothermel,et al.  Test case prioritization: an empirical study , 1999, Proceedings IEEE International Conference on Software Maintenance - 1999 (ICSM'99). 'Software Maintenance for Business Change' (Cat. No.99CB36360).

[15]  John T. Stasko,et al.  Visualization of test information to assist fault localization , 2002, ICSE '02.

[16]  Ondrej Lhoták,et al.  Comparing call graphs , 2007, PASTE '07.

[17]  James R. Larus,et al.  Efficient path profiling , 1996, Proceedings of the 29th Annual IEEE/ACM International Symposium on Microarchitecture. MICRO 29.

[18]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[19]  David F. Bacon,et al.  Fast and effective optimization of statically typed object-oriented languages , 1997 .

[20]  Sigrid Eldh.  Software Testing Techniques , 2007 .

[21]  Tao Xie,et al.  Test input reduction for result inspection to facilitate fault localization , 2010, Automated Software Engineering.

[22]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[23]  Mary Jean Harrold,et al.  Re-computing Coverage Information to Assist Regression Testing , 2007, 2007 IEEE International Conference on Software Maintenance.

[24]  Laurie A. Williams,et al.  System test case prioritization of new and regression test cases , 2005, 2005 International Symposium on Empirical Software Engineering, 2005..

[25]  Steven P. Reiss,et al.  Fault localization with nearest neighbor queries , 2003, 18th IEEE International Conference on Automated Software Engineering, 2003. Proceedings..

[26]  Bogdan Korel,et al.  Application of system models in regression test suite prioritization , 2008, 2008 IEEE International Conference on Software Maintenance.

[27]  Gregg Rothermel,et al.  On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques , 2006, IEEE Transactions on Software Engineering.

[28]  David F. Bacon,et al.  Fast static analysis of C++ virtual function calls , 1996, OOPSLA '96.

[29]  Jens Palsberg,et al.  Scalable propagation-based call graph construction algorithms , 2000, OOPSLA '00.

[30]  Neelam Gupta,et al.  Improving Fault Detection Capability by Selectively Retaining Test Cases during Test Suite Reduction , 2007, IEEE Transactions on Software Engineering.

[31]  David Notkin,et al.  An empirical study of static call graph extractors , 1998, TSEM.

[32]  Adam A. Porter,et al.  A history-based test prioritization technique for regression testing in resource constrained environments , 2002, ICSE '02.

[33]  Olin Shivers,et al.  Control-flow analysis of higher-order languages of taming lambda , 1991 .

[34]  Gregg Rothermel,et al.  Test Case Prioritization: A Family of Empirical Studies , 2002, IEEE Trans. Software Eng..

[35]  David Grove,et al.  Call graph construction in object-oriented languages , 1997, OOPSLA '97.

[36]  Alessandro Orso,et al.  MATRIX: Maintenance-Oriented Testing Requirements Identifier and Examiner , 2006, Testing: Academic & Industrial Conference - Practice And Research Techniques (TAIC PART'06).

[37]  Gregg Rothermel,et al.  Empirical studies of test case prioritization in a JUnit testing environment , 2004, 15th International Symposium on Software Reliability Engineering.

[38]  Jianjun Zhao,et al.  Test Case Prioritization Based on Analysis of Program Structure , 2008, 2008 15th Asia-Pacific Software Engineering Conference.

[39]  Atif M. Memon,et al.  Developing a Single Model and Test Prioritization Strategies for Event-Driven Software , 2011, IEEE Transactions on Software Engineering.

[40]  Gregg Rothermel,et al.  A controlled experiment assessing test case prioritization techniques via mutation faults , 2005, 21st IEEE International Conference on Software Maintenance (ICSM'05).

[41]  Gregg Rothermel,et al.  The Effects of Time Constraints on Test Case Prioritization: A Series of Controlled Experiments , 2010, IEEE Transactions on Software Engineering.

[42]  Tao Xie,et al.  Time-aware test-case prioritization using integer linear programming , 2009, ISSTA.

[43]  Mary Jean Harrold,et al.  Test-Suite Reduction and Prioritization for Modified Condition/Decision Coverage , 2003, IEEE Trans. Software Eng..

[44]  Neelam Gupta,et al.  Test Case Prioritization Using Relevant Slices , 2006, 30th Annual International Computer Software and Applications Conference (COMPSAC'06).

[45]  Lu Zhang,et al.  Prioritizing JUnit test cases in absence of coverage information , 2009, 2009 IEEE International Conference on Software Maintenance.