Improving Multi-Objective Test Case Selection by Injecting Diversity in Genetic Algorithms

A way to reduce the cost of regression testing consists of selecting or prioritizing subsets of test cases from a test suite according to some criteria. Besides greedy algorithms, cost cognizant additional greedy algorithms, multi-objective optimization algorithms, and multi-objective genetic algorithms (MOGAs), have also been proposed to tackle this problem. However, previous studies have shown that there is no clear winner between greedy and MOGAs, and that their combination does not necessarily produce better results. In this paper we show that the optimality of MOGAs can be significantly improved by diversifying the solutions (sub-sets of the test suite) generated during the search process. Specifically, we introduce a new MOGA, coined as DIversity based Genetic Algorithm (DIV-GA), based on the mechanisms of orthogonal design and orthogonal evolution that increase diversity by injecting new orthogonal individuals during the search process. Results of an empirical study conducted on eleven programs show that DIV-GA outperforms both greedy algorithms and the traditional MOGAs from the optimality point of view. Moreover, the solutions (sub-sets of the test suite) provided by DIV-GA are able to detect more faults than the other algorithms, while keeping the same test execution cost.

[1]  Gregg Rothermel,et al.  Empirical studies of test case prioritization in a JUnit testing environment , 2004, 15th International Symposium on Software Reliability Engineering.

[2]  Gregg Rothermel,et al.  Empirical Studies of a Safe Regression Test Selection Technique , 1998, IEEE Trans. Software Eng..

[3]  Gregg Rothermel,et al.  Empirical studies of test‐suite reduction , 2002, Softw. Test. Verification Reliab..

[4]  Douglas R. Stinson,et al.  Combinatorial designs: constructions and analysis , 2003, SIGA.

[5]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[6]  Joseph Robert Horgan,et al.  Effect of Test Set Minimization on Fault Detection Effectiveness , 1995, 1995 17th International Conference on Software Engineering.

[7]  Anne Auger,et al.  Theory of the hypervolume indicator: optimal μ-distributions and the choice of the reference point , 2009, FOGA '09.

[8]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[9]  P. Lachenbruch.  Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[10]  Atif M. Memon,et al.  A Uniform Representation of Hybrid Criteria for Regression Testing , 2013, IEEE Transactions on Software Engineering.

[11]  Lionel C. Briand,et al.  A practical guide for using statistical tests to assess randomized algorithms in software engineering , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[12]  Shin Yoo.  A Novel Mask-Coding Representation for Set Cover Problems with Applications in Test Suite Minimisation , 2010, 2nd International Symposium on Search Based Software Engineering.

[13]  Susan Horwitz,et al.  Incremental program testing using program dependence graphs , 1993, POPL '93.

[14]  Li Mo,et al.  A Cluster-Based Orthogonal Multi-Objective Genetic Algorithm , 2009 .

[15]  Lionel C. Briand,et al.  Achieving scalable model-based testing through test case diversity , 2013, TSEM.

[16]  Emanuel Melachrinoudis,et al.  Bi-criteria models for all-uses test suite reduction , 2004, Proceedings. 26th International Conference on Software Engineering.

[17]  Mark Harman.  Making the Case for MORTO: Multi Objective Regression Test Optimization , 2011, 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.

[18]  D. E. Goldberg,et al.  Genetic Algorithms in Search , 1989 .

[19]  Gary B. Lamont,et al.  Evolutionary Algorithms for Solving Multi-Objective Problems (Genetic and Evolutionary Computation) , 2006 .

[20]  Qingfu Zhang,et al.  An orthogonal genetic algorithm for multimedia multicast routing , 1999, IEEE Trans. Evol. Comput..

[21]  Lionel C. Briand,et al.  Empirical Investigation of the Effects of Test Suite Properties on Similarity-Based Test Case Selection , 2011, 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation.

[22]  Ernesto Benini,et al.  Genetic Diversity as an Objective in Multi-Objective Evolutionary Algorithms , 2003, Evolutionary Computation.

[23]  Myra B. Cohen,et al.  Constructing Interaction Test Suites for Highly-Configurable Systems in the Presence of Constraints: A Greedy Approach , 2008, IEEE Transactions on Software Engineering.

[24]  Tsong Yueh Chen,et al.  Adaptive Random Testing: The ART of test case diversity , 2010, J. Syst. Softw..

[25]  Samir W. Mahfoud.  Niching methods for genetic algorithms , 1996 .

[26]  Renée C. Bryce,et al.  A framework of greedy methods for constructing interaction test suites , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[27]  Kiyoshi Tanaka,et al.  Selection, Drift, Recombination, and Mutation in Multiobjective Evolutionary Algorithms on Scalable MNK-Landscapes , 2005, EMO.

[28]  Gregg Rothermel,et al.  Cost-cognizant Test Case Prioritization , 2006 .

[29]  Mark Harman,et al.  Using hybrid algorithm for Pareto efficient multi-objective test suite minimisation , 2010, J. Syst. Softw..

[30]  Vahid Garousi,et al.  A replicated survey of software testing practices in the Canadian province of Alberta: What has changed from 2004 to 2009? , 2010, J. Syst. Softw..

[31]  Amitabh Srivastava,et al.  Effectively prioritizing tests in development environment , 2002, ISSTA '02.

[32]  Nathaniel E. Helwig,et al.  An Introduction to Linear Algebra , 2006 .

[33]  Neelam Gupta,et al.  Test suite reduction with selective redundancy , 2005, 21st IEEE International Conference on Software Maintenance (ICSM'05).

[34]  Jasbir S. Arora,et al.  Survey of multi-objective optimization methods for engineering , 2004 .

[35]  Lionel C. Briand,et al.  Reducing the Cost of Model-Based Testing through Test Case Diversity , 2010, ICTSS.

[36]  Paolo Tonella,et al.  Orthogonal exploration of the search space in evolutionary test case generation , 2013, ISSTA.

[37]  Mark Harman,et al.  Search Algorithms for Regression Test Case Prioritization , 2007, IEEE Transactions on Software Engineering.

[38]  Mario Köppen,et al.  Substitute Distance Assignments in NSGA-II for Handling Many-objective Optimization Problems , 2007, EMO.

[39]  Atif M. Memon,et al.  Call stack coverage for test suite reduction , 2005, 21st IEEE International Conference on Software Maintenance (ICSM'05).

[40]  Tomoyuki Hiroyasu,et al.  Diversity Maintenance Mechanism for Multi-Objective Genetic Algorithms Using Clustering and Network Inversion , 2008, PPSN.

[41]  Gregg Rothermel,et al.  An empirical study of the effects of minimization on the fault detection capabilities of test suites , 1998, Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272).

[42]  Per Runeson,et al.  A survey of unit testing practices , 2006, IEEE Software.

[43]  Tsong Yueh Chen,et al.  Dividing Strategies for the Optimization of a Test Suite , 1996, Inf. Process. Lett..

[44]  Kaisa Miettinen,et al.  On initial populations of a genetic algorithm for continuous optimization problems , 2007, J. Glob. Optim..

[45]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[46]  Sik-Sang Yau,et al.  METHOD FOR REVALIDATING MODIFIED PROGRAMS IN THE MAINTENANCE PHASE. , 1987 .

[47]  S. Holm.  A Simple Sequentially Rejective Multiple Test Procedure , 1979 .

[48]  Aravind Seshadri,et al.  A FAST ELITIST MULTIOBJECTIVE GENETIC ALGORITHM: NSGA-II , 2000 .

[49]  Antonia Bertolino,et al.  Using Spanning Sets for Coverage Testing , 2003, IEEE Trans. Software Eng..

[50]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[51]  Marjan Mernik,et al.  Exploration and exploitation in evolutionary algorithms: A survey , 2013, CSUR.

[52]  Gregg Rothermel,et al.  Test Case Prioritization: A Family of Empirical Studies , 2002, IEEE Trans. Software Eng..

[53]  P. Gács,et al.  Algorithms , 1992 .

[54]  Andrea De Lucia,et al.  On the role of diversity measures for multi-objective test case selection , 2012, 2012 7th International Workshop on Automation of Software Test (AST).

[55]  Gregg Rothermel,et al.  Analyzing Regression Test Selection Techniques , 1996, IEEE Trans. Software Eng..

[56]  Rajiv Gupta,et al.  A methodology for controlling the size of a test suite , 1990, Proceedings. Conference on Software Maintenance 1990.

[57]  R. K. Ursem.  Multi-objective Optimization using Evolutionary Algorithms , 2009 .

[58]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[59]  David J. Groggel,et al.  Practical Nonparametric Statistics , 2000, Technometrics.

[60]  Andrea De Lucia,et al.  Estimating the evolution direction of populations to improve genetic algorithms , 2012, GECCO '12.

[61]  Lothar Thiele,et al.  The Hypervolume Indicator Revisited: On the Design of Pareto-compliant Indicators Via Weighted Integration , 2007, EMO.

[62]  George Mason,et al.  Procedures for Reducing the Size of Coverage-based Test Sets , 1995 .

[63]  Mark Harman,et al.  Pareto efficient multi-objective test case selection , 2007, ISSTA '07.

[64]  Gregg Rothermel,et al.  A safe, efficient algorithm for regression test selection , 1993, 1993 Conference on Software Maintenance.

[65]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[66]  Laurie A. Williams,et al.  System test case prioritization of new and regression test cases , 2005, 2005 International Symposium on Empirical Software Engineering, 2005..

[67]  R. A. Groeneveld,et al.  Practical Nonparametric Statistics (2nd ed). , 1981 .

[68]  Mark Harman,et al.  Highly Scalable Multi Objective Test Suite Minimisation Using Graphics Cards , 2011, SSBSE.

[69]  C. Borror.  Practical Nonparametric Statistics, 3rd Ed. , 2001 .

[70]  Li Zhang,et al.  Genetic Algorithm Based on the Orthogonal Design for Multidimensional Knapsack Problems , 2006, ICNC.

[71]  Georges R. Harik,et al.  Finding Multimodal Solutions Using Restricted Tournament Selection , 1995, ICGA.

[72]  A. E. Eiben,et al.  On Evolutionary Exploration and Exploitation , 1998, Fundam. Informaticae.

[73]  Gregg Rothermel,et al.  Incorporating varying test costs and fault severities into test case prioritization , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[74]  Peter R. Nelson,et al.  Design and Analysis of Experiments, 3rd Ed. , 1991 .

[75]  Mary Lou Soffa,et al.  TimeAware test suite prioritization , 2006, ISSTA '06.

[76]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.