Using hybrid algorithm for Pareto efficient multi-objective test suite minimisation

Test suite minimisation techniques seek to reduce the effort required for regression testing by selecting a subset of test suites. In previous work, the problem has been considered as a single-objective optimisation problem. However, real world regression testing can be a complex process in which multiple testing criteria and constraints are involved. This paper presents the concept of Pareto efficiency for the test suite minimisation problem. The Pareto-efficient approach is inherently capable of dealing with multiple objectives, providing the decision maker with a group of solutions that are not dominated by each other. The paper illustrates the benefits of Pareto efficient multi-objective test suite minimisation with empirical studies of two and three objective formulations, in which multiple objectives such as coverage and past fault-detection history are considered. The paper utilises a hybrid, multi-objective genetic algorithm that combines the efficient approximation of the greedy approach with the capability of population based genetic algorithm to produce higher-quality Pareto fronts.

[1]  Gregg Rothermel,et al.  Cost-cognizant Test Case Prioritization , 2006 .

[2]  Nicholas Nethercote,et al.  Valgrind: A Program Supervision Framework , 2003, RV@CAV.

[3]  Hareton K. N. Leung,et al.  Insights into regression testing (software testing) , 1989, Proceedings. Conference on Software Maintenance - 1989.

[4]  Shigeru Obayashi,et al.  Multidisciplinary design optimization of aircraft wing planform based on evolutionary algorithms , 1998, SMC'98 Conference Proceedings. 1998 IEEE International Conference on Systems, Man, and Cybernetics (Cat. No.98CH36218).

[5]  Mark Gershon,et al.  Techniques for multiobjective decision making in systems management , 1986 .

[6]  Joseph Feller.  Meeting challenges and surviving success: the 2nd workshop on open source software engineering , 2002, Proceedings of the 24th International Conference on Software Engineering. ICSE 2002.

[7]  Alistair Cockburn,et al.  Agile Software Development , 2001 .

[8]  W. Eric Wong,et al.  Effect of test set minimization on fault detection effectiveness , 1998 .

[9]  Tomasz Arciszewski,et al.  Multiobjective evolutionary design of steel structures in tall buildings , 2004 .

[10]  Kalyanmoy Deb,et al.  A Fast Elitist Non-dominated Sorting Genetic Algorithm for Multi-objective Optimisation: NSGA-II , 2000, PPSN.

[11]  M. Dufwenberg.  Game theory. , 2011, Wiley interdisciplinary reviews. Cognitive science.

[12]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[13]  Gregg Rothermel,et al.  Empirical studies of test case prioritization in a JUnit testing environment , 2004, 15th International Symposium on Software Reliability Engineering.

[14]  David S. Johnson,et al.  Computers and Intractability: A Guide to the Theory of NP-Completeness , 1978 .

[15]  Walter F. Tichy,et al.  Proceedings 25th International Conference on Software Engineering , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[16]  Susan Horwitz,et al.  Incremental program testing using program dependence graphs , 1993, POPL '93.

[17]  Gregg Rothermel,et al.  Test case prioritization: an empirical study , 1999, Proceedings IEEE International Conference on Software Maintenance - 1999 (ICSM'99). 'Software Maintenance for Business Change' (Cat. No.99CB36360).

[18]  David S. Johnson,et al.  Approximation algorithms for combinatorial problems , 1973, STOC.

[19]  Graham Kendall,et al.  Hyper-Heuristics: An Emerging Direction in Modern Search Technology , 2003, Handbook of Metaheuristics.

[20]  Kim Fung Man,et al.  Multiobjective Optimization , 2011, IEEE Microwave Magazine.

[21]  Mark Harman,et al.  Pareto efficient multi-objective test case selection , 2007, ISSTA '07.

[22]  Gregg Rothermel,et al.  Analyzing Regression Test Selection Techniques , 1996, IEEE Trans. Software Eng..

[23]  Thomas J. Ostrand,et al.  Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria , 1994, Proceedings of 16th International Conference on Software Engineering.

[24]  Nicholas Nethercote,et al.  Valgrind: a framework for heavyweight dynamic binary instrumentation , 2007, PLDI '07.

[25]  David W. Binkley,et al.  Reducing the cost of regression testing by semantics guided test case selection , 1995, Proceedings of International Conference on Software Maintenance.

[26]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[27]  Joseph Robert Horgan,et al.  Effect of Test Set Minimization on Fault Detection Effectiveness , 1995, 1995 17th International Conference on Software Engineering.

[28]  Alistair Cockburn,et al.  Agile Software Development: The Business of Innovation , 2001, Computer.

[29]  Gregg Rothermel,et al.  Proceedings of the 2004 ACM SIGSOFT international symposium on Software testing and analysis , 2004 .

[30]  Rajiv Gupta,et al.  A methodology for controlling the size of a test suite , 1990, Proceedings. Conference on Software Maintenance 1990.

[31]  R. K. Ursem.  Multi-objective Optimization using Evolutionary Algorithms , 2009 .

[32]  Sik-Sang Yau,et al.  METHOD FOR REVALIDATING MODIFIED PROGRAMS IN THE MAINTENANCE PHASE. , 1987 .

[33]  Christian Fonteix,et al.  Multicriteria optimization of an emulsion polymerization process , 2004, Eur. J. Oper. Res..

[34]  P. Siarry,et al.  Multiobjective Optimization: Principles and Case Studies , 2004 .

[35]  Mark Harman,et al.  Search Algorithms for Regression Test Case Prioritization , 2007, IEEE Transactions on Software Engineering.

[36]  Gregg Rothermel,et al.  An empirical study of the effects of minimization on the fault detection capabilities of test suites , 1998, Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272).

[37]  Gregg Rothermel,et al.  Incorporating varying test costs and fault severities into test case prioritization , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[38]  Mary Lou Soffa,et al.  TimeAware test suite prioritization , 2006, ISSTA '06.

[39]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[40]  Gary B. Lamont,et al.  Evolutionary Algorithms for Solving Multi-Objective Problems , 2002, Genetic Algorithms and Evolutionary Computation.