Pareto efficient multi-objective test case selection

Previous work has treated test case selection as a single objective optimisation problem. This paper introduces the concept of Pareto efficiency to test case selection. The Pareto efficient approach takes multiple objectives such as code coverage, past fault-detection history and execution cost, and constructs a group of non-dominating, equivalently optimal test case subsets. The paper describes the potential bene?ts of Pareto efficient multi-objective test case selection, illustrating with empirical studies of two and three objective formulations.

[1]  Kalyanmoy Deb,et al.  MULTI-OBJECTIVE FUNCTION OPTIMIZATION USING NON-DOMINATED SORTING GENETIC ALGORITHMS , 1994 .

[2]  Gregg Rothermel,et al.  Cost-cognizant Test Case Prioritization , 2006 .

[3]  Sik-Sang Yau,et al.  METHOD FOR REVALIDATING MODIFIED PROGRAMS IN THE MAINTENANCE PHASE. , 1987 .

[4]  Nicholas Nethercote,et al.  Valgrind: A Program Supervision Framework , 2003, RV@CAV.

[5]  W. Eric Wong,et al.  Effect of test set minimization on fault detection effectiveness , 1998 .

[6]  Susan Horwitz,et al.  Incremental program testing using program dependence graphs , 1993, POPL '93.

[7]  Mark Harman,et al.  Search-based software engineering , 2001, Inf. Softw. Technol..

[8]  Marco Laumanns,et al.  SPEA2: Improving the Strength Pareto Evolutionary Algorithm For Multiobjective Optimization , 2002 .

[9]  Paolo Tonella,et al.  Using the Case-Based Ranking Methodology for Test Case Prioritization , 2006, 2006 22nd IEEE International Conference on Software Maintenance.

[10]  Lothar Thiele,et al.  Multiobjective evolutionary algorithms: a comparative case study and the strength Pareto approach , 1999, IEEE Trans. Evol. Comput..

[11]  Mark Harman,et al.  Meta-heuristic Search Algorithms for Regression Test Case Prioritization , 2007 .

[12]  P. Siarry,et al.  Multiobjective Optimization: Principles and Case Studies , 2004 .

[13]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[14]  Gary B. Lamont,et al.  Evolutionary Algorithms for Solving Multi-Objective Problems (Genetic and Evolutionary Computation) , 2006 .

[15]  Gregg Rothermel,et al.  Test case prioritization: an empirical study , 1999, Proceedings IEEE International Conference on Software Maintenance - 1999 (ICSM'99). 'Software Maintenance for Business Change' (Cat. No.99CB36360).

[16]  Mark Gershon,et al.  Techniques for multiobjective decision making in systems management , 1986 .

[17]  Gary B. Lamont,et al.  Evolutionary Algorithms for Solving Multi-Objective Problems , 2002, Genetic Algorithms and Evolutionary Computation.

[18]  Mary Lou Soffa,et al.  TimeAware test suite prioritization , 2006, ISSTA '06.

[19]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[20]  Joseph Robert Horgan,et al.  Effect of Test Set Minimization on Fault Detection Effectiveness , 1995, 1995 17th International Conference on Software Engineering.

[21]  David W. Binkley,et al.  Reducing the cost of regression testing by semantics guided test case selection , 1995, Proceedings of International Conference on Software Maintenance.

[22]  Mark Harman,et al.  Search Algorithms for Regression Test Case Prioritization , 2007, IEEE Transactions on Software Engineering.

[23]  Gregg Rothermel,et al.  An empirical study of the effects of minimization on the fault detection capabilities of test suites , 1998, Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272).

[24]  Kalyanmoy Deb,et al.  A Fast Elitist Non-dominated Sorting Genetic Algorithm for Multi-objective Optimisation: NSGA-II , 2000, PPSN.

[25]  Gregg Rothermel,et al.  Analyzing Regression Test Selection Techniques , 1996, IEEE Trans. Software Eng..

[26]  Thomas J. Ostrand,et al.  Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria , 1994, Proceedings of 16th International Conference on Software Engineering.

[27]  Rajiv Gupta,et al.  A methodology for controlling the size of a test suite , 1990, Proceedings. Conference on Software Maintenance 1990.

[28]  R. K. Ursem.  Multi-objective Optimization using Evolutionary Algorithms , 2009 .

[29]  Gregg Rothermel,et al.  Test Case Prioritization: A Family of Empirical Studies , 2002, IEEE Trans. Software Eng..

[30]  Mark G. Sobell.  A Practical Guide to the UNIX System , 1994 .

[31]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[32]  Martin J. Oates,et al.  The Pareto Envelope-Based Selection Algorithm for Multi-objective Optimisation , 2000, PPSN.

[33]  Goldberg,et al.  Genetic algorithms , 1993, Robust Control Systems with Genetic Algorithms.