A systematic mapping addressing Hyper-Heuristics within Search-based Software Testing

Abstract Context Search-based Software Testing (SBST) is a research field where testing a software product is formulated as an optimization problem. It is an active sub-area of Search-based Software Engineering (SBSE) where many studies have been published and some reviews have been carried out. The majority of studies in SBST has been adopted meta-heuristics while hyper-heuristics have a long way to go. Moreover, there is still a lack of studies to perceive the state-of-the-art of the use of hyper-heuristics within SBST. Objective The objective of this work is to investigate the adoption of hyper-heuristics for Software Testing highlighting the current efforts and identifying new research directions. Method A Systematic mapping study was carried out with 5 research questions considering papers published up to may/2019, and 4 different bases. The research questions aims to find out, among other things, what are the hyper-heuristics used in the context of Software Testing, for what problems hyper-heuristics have been applied, and what are the objective functions in the scope of Software Testing. Results A total of 734 studies were found via the search strings and 164 articles were related to Software Testing. However, from these, only 26 papers were actually in accordance with the scope of this research and 3 more papers were considered due to snowballing or expert’s suggestion, totalizing 29 selected papers. Few different problems and application domains where hyper-heuristics have been considered were identified. Conclusion Differently from other communities (Operational Research, Artificial Intelligence), SBST has little explored the benefits of hyper-heuristics which include generalization and less difficulty in parameterization. Hence, it is important to further investigate this area in order to alleviate the effort of practitioners to use such an approach in their testing activities.

[1]  Gian Mauricio Fritsche,et al.  A Hyper-Heuristic for the Multi-Objective Integration and Test Order Problem , 2015, GECCO.

[2]  Phil McMinn,et al.  Search-Based Software Testing: Past, Present and Future , 2011, 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.

[3]  Bestoun S. Ahmed,et al.  An efficient strategy for covering array construction with fuzzy logic-based adaptive swarm optimization for software testing use , 2015, Expert Syst. Appl..

[4]  L. Hedges,et al.  The Handbook of Research Synthesis , 1995 .

[5]  Silvia Regina Vergilio,et al.  Multiple objective test set selection for software product line testing: evaluating different preference-based algorithms , 2018, SBES.

[6]  Yuanyuan Zhang,et al.  Achievements, Open Problems and Challenges for Search Based Software Testing , 2015, 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST).

[7]  Prabhat Kumar,et al.  An extensive evaluation of search-based software testing: a review , 2017, Soft Computing.

[8]  James Miller,et al.  Automatic test data generation using genetic algorithm and program dependence graphs , 2006, Inf. Softw. Technol..

[9]  Myra B. Cohen,et al.  Learning Combinatorial Interaction Test Generation Strategies Using Hyperheuristic Search , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[10]  Siti Hafizah Ab Hamid,et al.  The experimental applications of search-based techniques for model-based testing: Taxonomy and systematic literature review , 2016, Appl. Soft Comput..

[11]  José Torres-Jiménez,et al.  Construction of Mixed Covering Arrays of Variable Strength Using a Tabu Search Approach , 2010, COCOA.

[12]  Maninder Singh,et al.  A systematic review of approaches for testing concurrent programs , 2016, Concurr. Comput. Pract. Exp..

[13]  Ender Özcan,et al.  A Learning Automata-Based Multiobjective Hyper-Heuristic , 2019, IEEE Transactions on Evolutionary Computation.

[14]  James Miller,et al.  Black-Box String Test Case Generation through a Multi-Objective Optimization , 2016, IEEE Transactions on Software Engineering.

[15]  Victor I. Chang,et al.  A survey on Test Suite Reduction frameworks and tools , 2016, Int. J. Inf. Manag..

[16]  Mark Harman,et al.  Search-based software engineering , 2001, Inf. Softw. Technol..

[17]  Graham Kendall,et al.  A Tabu Search hyper-heuristic strategy for t-way test suite generation , 2016, Appl. Soft Comput..

[18]  Graham Kendall,et al.  A Hyperheuristic Approach to Scheduling a Sales Summit , 2000, PATAT.

[19]  Gian Mauricio Fritsche,et al.  A multi-objective and evolutionary hyper-heuristic applied to the Integration and Test Order Problem , 2017, Appl. Soft Comput..

[20]  Graham Kendall,et al.  An experimental study of hyper-heuristic selection and acceptance mechanism for combinatorial t-way test suite generation , 2017, Inf. Sci..

[21]  Wasif Afzal,et al.  A systematic review of search-based testing for non-functional system properties , 2009, Inf. Softw. Technol..

[22]  Andreas Zeller,et al.  Mutation-Driven Generation of Unit Tests and Oracles , 2012, IEEE Trans. Software Eng..

[23]  Ender Özcan,et al.  Improving performance of a hyper-heuristic using a multilayer perceptron for vehicle routing , 2015 .

[24]  Zheng Li,et al.  Concrete hyperheuristic framework for test case prioritization , 2018, J. Softw. Evol. Process..

[25]  Kamal Z. Zamli,et al.  A Parameter Free Choice Function Based Hyper-Heuristic Strategy for Pairwise Test Generation , 2017, 2017 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C).

[26]  Aurora Trinidad Ramirez Pozo,et al.  Product selection based on upper confidence bound MOEA/D-DRA for testing software product lines , 2016, 2016 IEEE Congress on Evolutionary Computation (CEC).

[27]  Yuanyuan Zhang,et al.  Search-based software engineering: Trends, techniques and applications , 2012, CSUR.

[28]  Joachim Wegener,et al.  Applying particle swarm optimization to software testing , 2007, GECCO '07.

[29]  Saïd Salhi,et al.  Hyper-heuristic approaches for the response time variability problem , 2011, Eur. J. Oper. Res..

[30]  Yuanyuan Zhang,et al.  An Empirical Study of Meta- and Hyper-Heuristic Search for Multi-Objective Release Planning , 2018, ACM Trans. Softw. Eng. Methodol..

[31]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[32]  Hakjoo Oh,et al.  Automatically Generating Search Heuristics for Concolic Testing , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[33]  Grigore Albeanu,et al.  Nature-inspired Approaches in Software Faults Identification and Debugging , 2016 .

[34]  Aurora Trinidad Ramirez Pozo,et al.  Grammatical Evolution for the Multi-Objective Integration and Test Order Problem , 2016, GECCO.

[35]  Kai Petersen,et al.  Systematic Mapping Studies in Software Engineering , 2008, EASE.

[36]  Erik M. Fredericks.  An Empirical Analysis of the Mutation Operator for Run-Time Adaptive Testing in Self-Adaptive Systems , 2018, 2018 IEEE/ACM 11th International Workshop on Search-Based Software Testing (SBST).

[37]  Silvia Regina Vergilio,et al.  A systematic review on search-based refactoring , 2017, Inf. Softw. Technol..

[38]  Eunseok Lee,et al.  A new hybrid algorithm for software fault localization , 2015, IMCOM.

[39]  Yue Jia.  Hyperheuristic Search for SBST , 2015, 2015 IEEE/ACM 8th International Workshop on Search-Based Software Testing.

[40]  Domagoj Jakobovic,et al.  Genetic Programming Heuristics for Multiple Machine Scheduling , 2007, EuroGP.

[41]  Kent McClymont,et al.  Markov chain hyper-heuristic (MCHH): an online selective hyper-heuristic for multi-objective continuous problems , 2011, GECCO '11.

[42]  María Cristina Riff,et al.  DVRP: a hard dynamic combinatorial optimisation problem tackled by an evolutionary hyper-heuristic , 2010, J. Heuristics.

[43]  Giovani Guizzo,et al.  A pattern-driven solution for designing multi-objective evolutionary algorithms , 2018, Natural Computing.

[44]  Aurora Trinidad Ramirez Pozo,et al.  Hyper-Heuristic Based Product Selection for Software Product Line Testing , 2017, IEEE Computational Intelligence Magazine.

[45]  P. J. Nikumbh,et al.  Pair-wise Testing Using Simulated Annealing , 2012 .

[46]  D. Jeya Mala,et al.  Quality improvement and optimization of test cases: a hybrid genetic algorithm based approach , 2010, SOEN.

[47]  Riccardo Poli,et al.  A histogram-matching approach to the evolution of bin-packing strategies , 2007, 2007 IEEE Congress on Evolutionary Computation.

[48]  Xiong Xu,et al.  Boosting Search Based Software Testing by Using Ensemble Methods , 2018, 2018 IEEE Congress on Evolutionary Computation (CEC).

[49]  Aurora Trinidad Ramirez Pozo,et al.  Deriving products for variability test of Feature Models with a hyper-heuristic approach , 2016, Appl. Soft Comput..

[50]  Lionel C. Briand,et al.  Achieving scalable model-based testing through test case diversity , 2013, TSEM.

[51]  Graham Kendall,et al.  A multi-objective hyper-heuristic based on choice function , 2014, Expert Syst. Appl..

[52]  Mark Harman,et al.  Pareto efficient multi-objective test case selection , 2007, ISSTA '07.

[53]  Upulee Kanewala,et al.  Automated Test Oracles: State of the Art, Taxonomies, and Trends , 2015, Adv. Comput..

[54]  Tony Gorschek,et al.  Requirements engineering for safety-critical systems: A systematic literature review , 2016, Inf. Softw. Technol..

[55]  Yvan Labiche,et al.  Multi-objective Construction of an Entire Adequate Test Suite for an EFSM , 2014, 2014 IEEE 25th International Symposium on Software Reliability Engineering.

[56]  Enrique Alba,et al.  Evolutionary algorithms for the multi‐objective test data generation problem , 2012, Softw. Pract. Exp..

[57]  Mosab Bazargani,et al.  A Hyper-heuristic for Multi-objective Integration and Test Ordering in Google Guava , 2017, SSBSE.

[58]  José Javier Dolado,et al.  A tabu search algorithm for structural software testing , 2008, Comput. Oper. Res..

[59]  Jackson A. Prado Lima,et al.  Automatic Generation of Search-Based Algorithms Applied to the Feature Testing of Software Product Lines , 2017, SBES'17.

[60]  Anju Saha,et al.  A review of applications of search based software engineering techniques in last decade , 2016, 2016 5th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO).

[61]  Myra B. Cohen,et al.  Practical Combinatorial Interaction Testing: Empirical Findings on Efficiency and Early Fault Detection , 2015, IEEE Transactions on Software Engineering.

[62]  Jackson A. Prado Lima,et al.  A Multi-objective optimization approach for selection of second order mutant generation strategies , 2017, SAST.

[63]  Tsong Yueh Chen,et al.  Adaptive Random Testing: The ART of test case diversity , 2010, J. Syst. Softw..

[64]  Anjana Gosain,et al.  A comparative analysis of ant colony optimization for its applications into software testing , 2014, 2014 Innovative Applications of Computational Intelligence on Power, Energy and Controls with their impact on Humanity (CIPECH).

[65]  Giuliano Antoniol,et al.  Automatic mutation test input data generation via ant colony , 2007, GECCO '07.

[66]  Michel Gendreau,et al.  Hyper-heuristics: a survey of the state of the art , 2013, J. Oper. Res. Soc..

[67]  Dayang N. A. Jawawi,et al.  Test case prioritization approaches in regression testing: A systematic literature review , 2018, Inf. Softw. Technol..

[68]  Marco Laumanns,et al.  SPEA2: Improving the strength pareto evolutionary algorithm , 2001 .

[69]  Myra B. Cohen.  The evolutionary landscape of SBST: a 10 year perspective , 2017 .

[70]  Ender Özcan,et al.  Multi-objective evolutionary algorithms and hyper-heuristics for wind farm layout optimisation , 2017 .

[71]  Aurora Trinidad Ramirez Pozo,et al.  Evaluating a Multi-objective Hyper-Heuristic for the Integration and Test Order Problem , 2015, 2015 Brazilian Conference on Intelligent Systems (BRACIS).

[72]  Myra B. Cohen,et al.  An orchestrated survey of methodologies for automated software test case generation , 2013, J. Syst. Softw..

[73]  Silvia Regina Vergilio,et al.  Incorporating User Preferences in a Software Product Line Testing Hyper-Heuristic Approach , 2018, CEC.

[74]  Simone do Rocio Senger de Souza,et al.  A systematic review on search based mutation testing , 2017, Inf. Softw. Technol..

[75]  Luca Maria Gambardella,et al.  Handling constraints in combinatorial interaction testing in the presence of multi objective particle swarm and multithreading , 2017, Inf. Softw. Technol..