A hybrid particle swarm optimization and harmony search algorithm approach for multi-objective test case selection

Test case (TC) selection is considered a hard problem, due to the high number of possible combinations to consider. Search-based optimization strategies arise as a promising way to treat this problem, as they explore the space of possible solutions (subsets of TCs), seeking the solution that best satisfies the given test adequacy criterion. The TC subsets are evaluated by an objective function, which must be optimized. In particular, we focus on multi-objective optimization (MOO) search-based strategies, which are able to properly treat TC selection problems with more than one test adequacy criterion. In this paper, we proposed two MOO algorithms (BMOPSO-CDR and BMOPSO-CDRHS) and present experimental results comparing both with two baseline algorithms: NSGA-II and MBHS. The experiments covered both structural and functional testing scenarios. The results show better performance of the BMOPSO-CDRHS algorithm for almost of all experiments. Furthermore, the performance of the algorithms was not impacted by the type of testing being used. The hybridization indeed improved the performance of the MOO PSO used as baseline and the proposed hybrid algorithm demonstrated to be competitive compared with other MOO algorithms.

[1]  Zong Woo Geem,et al.  A New Heuristic Optimization Algorithm: Harmony Search , 2001, Simul..

[2]  Mary Lou Soffa,et al.  Interprocedual data flow testing , 1989 .

[3]  Mark Harman.  Making the Case for MORTO: Multi Objective Regression Test Optimization , 2011, 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.

[4]  Hans van Vliet,et al.  Software engineering - principles and practice , 1993 .

[5]  Nicholas Nethercote,et al.  Valgrind: A Program Supervision Framework , 2003, RV@CAV.

[6]  Xin Yao,et al.  Parallel Problem Solving from Nature PPSN VI , 2000, Lecture Notes in Computer Science.

[7]  Wilkerson de L. Andrade,et al.  LTS-BT: a tool to generate and select functional test cases for embedded systems , 2008, SAC '08.

[8]  Ricardo B. C. Prudêncio,et al.  A Constrained Particle Swarm Optimization Approach for Test Case Selection , 2010, SEKE.

[9]  Lionel C. Briand,et al.  Automating impact analysis and regression test selection based on UML designs , 2002, International Conference on Software Maintenance, 2002. Proceedings..

[10]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[11]  Mark Harman,et al.  Faster Fault Finding at Google Using Multi Objective Regression Test Optimisation , 2011 .

[12]  Russell C. Eberhart,et al.  Comparison between Genetic Algorithms and Particle Swarm Optimization , 1998, Evolutionary Programming.

[13]  Marion G. Ceruti,et al.  Supporting C2 Research and Evaluation: An Infrastructure and its Potential Impact , 2011 .

[14]  Carlos A. Coello Coello,et al.  Handling multiple objectives with particle swarm optimization , 2004, IEEE Transactions on Evolutionary Computation.

[15]  Ladan Tahvildari,et al.  Size-Constrained Regression Test Case Selection Using Multicriteria Optimization , 2012, IEEE Transactions on Software Engineering.

[16]  Kalyanmoy Deb,et al.  A Fast Elitist Non-dominated Sorting Genetic Algorithm for Multi-objective Optimisation: NSGA-II , 2000, PPSN.

[17]  Mauro Pezzè,et al.  Software testing and analysis - process, principles and techniques , 2007 .

[18]  Arnaud Gotlieb,et al.  Minimizing test suites in software product lines using weight-based genetic algorithms , 2013, GECCO '13.

[19]  Joseph Robert Horgan,et al.  Incremental regression testing , 1993, 1993 Conference on Software Maintenance.

[20]  Sik-Sang Yau,et al.  METHOD FOR REVALIDATING MODIFIED PROGRAMS IN THE MAINTENANCE PHASE. , 1987 .

[21]  Mark Harman,et al.  Using hybrid algorithm for Pareto efficient multi-objective test suite minimisation , 2010, J. Syst. Softw..

[22]  Gopalaswamy Ramesh,et al.  Software Testing: Principles and Practices , 2005 .

[23]  Eduardo Aranha,et al.  Model Based Test Generation : An Industrial Experience , 2007 .

[24]  Gregg Rothermel,et al.  A safe, efficient regression test selection technique , 1997, TSEM.

[25]  Mary Lou Soffa,et al.  A methodology for controlling the size of a test suite , 1993, TSEM.

[26]  M. P. Gupta,et al.  Multi-objective test suite minimisation using Quantum-inspired Multi-objective Differential Evolution Algorithm , 2012, 2012 IEEE International Conference on Computational Intelligence and Computing Research.

[27]  Mark Harman,et al.  Highly Scalable Multi Objective Test Suite Minimisation Using Graphics Cards , 2011, SSBSE.

[28]  Chin-Yu Huang,et al.  Analysis of test suite reduction with enhanced tie-breaking techniques , 2009, Inf. Softw. Technol..

[29]  Gustavo Augusto,et al.  A MULTI-OBJECTIVE APPROACH FOR THE REGRESSION TEST CASE SELECTION PROBLEM , 2009 .

[30]  Paul C. Jorgensen,et al.  Software Testing: A Craftsman's Approach , 1995 .

[31]  Minrui Fei,et al.  A Multi-Objective Binary Harmony Search Algorithm , 2011, ICSI.

[32]  Gary B. Lamont,et al.  Evolutionary Algorithms for Solving Multi-Objective Problems , 2002, Genetic Algorithms and Evolutionary Computation.

[33]  Paulo Borba,et al.  Using Process Simulation to Assess the Test Design Effort Reduction of a Model-Based Testing Approach , 2008, ICSP.

[34]  Mustafa Bozkurt.  Cost-aware pareto optimal test suite minimisation for service-centric systems , 2013, GECCO '13.

[35]  Mark Harman,et al.  GPGPU test suite minimisation: search based software engineering performance improvement using graphics cards , 2013, Empirical Software Engineering.

[36]  Patrícia Duarte de Lima Machado,et al.  On the use of a similarity function for test case selection in the context of model‐based testing , 2011, Softw. Test. Verification Reliab..

[37]  F. Osório,et al.  Journal of the Brazilian Computer Society , 2009 .

[38]  Ricardo B. C. Prudêncio,et al.  A Multi-objective Particle Swarm Optimization for Test Case Selection Based on Functional Requirements Coverage and Execution Effort , 2011, 2011 IEEE 23rd International Conference on Tools with Artificial Intelligence.

[39]  Phyllis G. Frankl,et al.  Empirical evaluation of the textual differencing regression testing technique , 1998, Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272).

[40]  Mark Harman,et al.  Pareto efficient multi-objective test case selection , 2007, ISSTA '07.

[41]  Brad Clement,et al.  Automated Test Case Selection for Flight Systems using Genetic Algorithms , 2010 .

[42]  Emanuel Melachrinoudis,et al.  Bi-criteria models for all-uses test suite reduction , 2004, Proceedings. 26th International Conference on Software Engineering.

[43]  Nicolae Goga,et al.  Test Selection, Trace Distance and Heuristics , 2002, TestCom.

[44]  Kalyanmoy Deb,et al.  Multi-objective optimization using evolutionary algorithms , 2001, Wiley-Interscience series in systems and optimization.

[45]  Thomas J. Ostrand,et al.  Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria , 1994, Proceedings of 16th International Conference on Software Engineering.

[46]  Rajiv Gupta,et al.  A methodology for controlling the size of a test suite , 1990, Proceedings. Conference on Software Maintenance 1990.

[47]  R. Prudêncio,et al.  Search based constrained test case selection using execution effort , 2013, Expert Syst. Appl..

[48]  George W. Irwin,et al.  Life System Modeling and Intelligent Computing , 2011 .

[49]  Ali Soleimani,et al.  A Binary Harmony Search Algorithm for Solving the Maximum Clique Problem , 2013 .

[50]  Andrea De Lucia,et al.  On the role of diversity measures for multi-objective test case selection , 2012, 2012 7th International Workshop on Automation of Software Test (AST).

[51]  James Kennedy,et al.  Particle swarm optimization , 2002, Proceedings of ICNN'95 - International Conference on Neural Networks.

[52]  Minrui Fei,et al.  A Discrete Harmony Search Algorithm , 2010 .

[53]  Rudolf Ramler,et al.  Economic perspectives in test automation: balancing automated and manual testing with opportunity cost , 2006, AST '06.

[54]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.