An Insight Into Test Case Optimization: Ideas and Trends With Future Perspectives

Software engineering comprises several disciplines. Testing, the subject of this paper, is an important phase which is still largely unpredictable, expensive, and ad hoc. Software testing has already been framed as an optimization problem, i.e., to solve problems in software testing various optimization techniques can be used. To adequately test the software, various rules called test adequacy criteria are used. In this paper, a review of optimization techniques used in domains, test case generation, selection, minimization, and prioritization of testing, has been presented. The review is conducted by selecting quality literature from journals, conferences, workshops, and symposium. The state-of-the-art issues in software testing have been addressed by analyzing the reviewed literature on the basis of the domain of testing, test adequacy criteria, and optimization technique used. After analysis of the considered literature, authors were able to conclude the limitations of existing work, list of traditional and recently proposed adequacy criteria as well as combinations of adequacy criteria for multi-objective optimization and optimization techniques which may be used in the optimization of software testing.

[1]  Myra B. Cohen,et al.  An orchestrated survey of methodologies for automated software test case generation , 2013, J. Syst. Softw..

[2]  Ladan Tahvildari,et al.  Size-Constrained Regression Test Case Selection Using Multicriteria Optimization , 2012, IEEE Transactions on Software Engineering.

[3]  Trong-The Nguyen,et al.  Hybrid Particle Swarm Optimization with Bat Algorithm , 2014, ICGEC.

[4]  Gaige Wang,et al.  A Novel Hybrid Bat Algorithm with Harmony Search for Global Numerical Optimization , 2013, J. Appl. Math..

[5]  Xin-She Yang,et al.  Optimal test sequence generation using firefly algorithm , 2013, Swarm Evol. Comput..

[6]  Gai-Ge Wang,et al.  An Effective Hybrid Firefly Algorithm with Harmony Search for Global Numerical Optimization , 2013, TheScientificWorldJournal.

[7]  Gordon Fraser,et al.  Achieving scalable mutation-based generation of whole test suites , 2015, Empirical Software Engineering.

[8]  Baowen Xu,et al.  Application of Genetic Algorithms in Software Testing , 2007 .

[9]  Ricardo B. C. Prudêncio,et al.  A Multi-objective Particle Swarm Optimization for Test Case Selection Based on Functional Requirements Coverage and Execution Effort , 2011, 2011 IEEE 23rd International Conference on Tools with Artificial Intelligence.

[10]  Yuanyuan Zhang,et al.  Achievements, Open Problems and Challenges for Search Based Software Testing , 2015, 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST).

[11]  Wasif Afzal,et al.  A systematic review of search-based testing for non-functional system properties , 2009, Inf. Softw. Technol..

[12]  Seyedali Mirjalili,et al.  Dragonfly algorithm: a new meta-heuristic optimization technique for solving single-objective, discrete, and multi-objective problems , 2015, Neural Computing and Applications.

[13]  Mark Harman,et al.  GPGPU test suite minimisation: search based software engineering performance improvement using graphics cards , 2013, Empirical Software Engineering.

[14]  Maral Azizi,et al.  Graphite: A Greedy Graph-Based Technique for Regression Test Case Prioritization , 2018, 2018 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW).

[15]  Mark Harman,et al.  A multi-objective approach to search-based test data generation , 2007, GECCO '07.

[16]  Manoj Kumar,et al.  Multi-Faceted Measurement Framework for Test Case Classification and Fitness Evaluation using Fuzzy Logic Based Approach , 2012 .

[17]  David L. Spooner,et al.  Automatic Generation of Floating-Point Test Data , 1976, IEEE Transactions on Software Engineering.

[18]  Gordon Fraser,et al.  Assessing and generating test sets in terms of behavioural adequacy , 2015, Softw. Test. Verification Reliab..

[19]  Stewart N. Weiss,et al.  Methods of comparing test data adequacy criteria , 1990, Proceedings., Fourteenth Annual International Computer Software and Applications Conference.

[20]  Shin Yoo,et al.  Diversity-Aware Mutation Adequacy Criterion for Improving Fault Detection Capability , 2016, 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[21]  Hong Zhu,et al.  Software unit test coverage and adequacy , 1997, ACM Comput. Surv..

[22]  Bestoun S. Ahmed.  Test case minimization approach using fault detection and combinatorial optimization techniques for configuration-aware structural testing , 2016 .

[23]  Manika Tyagi,et al.  Test case prioritization using multi objective particle swarm optimizer , 2014, 2014 International Conference on Signal Propagation and Computer Technology (ICSPCT 2014).

[24]  Shin Yoo,et al.  Empirical Evaluation of Mutation-based Test Prioritization Techniques , 2017, ArXiv.

[25]  Doo-Hwan Bae,et al.  A Theoretical Framework for Understanding Mutation-Based Testing Methods , 2016, 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST).

[26]  Jun Cheng,et al.  A Fine-Grained Parallel Multi-objective Test Case Prioritization on GPU , 2013, SSBSE.

[27]  James Miller,et al.  Black-Box String Test Case Generation through a Multi-Objective Optimization , 2016, IEEE Transactions on Software Engineering.

[28]  D. Jeya Mala,et al.  Quality improvement and optimization of test cases: a hybrid genetic algorithm based approach , 2010, SOEN.

[29]  Rajesh Kumar,et al.  Fuzzy entropy-based framework for multi-faceted test case classification and selection: an empirical study , 2014, IET Softw..

[30]  Farhad Soleimanian Gharehchopogh,et al.  A Novel Hybrid Artificial Immune System with Genetic Algorithm for Software Cost Estimation , 2014 .

[31]  Paolo Tonella,et al.  Automated Test Case Generation as a Many-Objective Optimisation Problem with Dynamic Selection of the Targets , 2018, IEEE Transactions on Software Engineering.

[32]  Shuai Wang,et al.  Search-Based Cost-Effective Test Case Selection within a Time Budget: An Empirical Study , 2016, GECCO.

[33]  Stephane Durocher,et al.  Exploring Test Suite Diversification and Code Coverage in Multi-Objective Test Case Selection , 2015, 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST).

[34]  Arun Sharma,et al.  An empirical evaluation of a three‐tier conduit framework for multifaceted test case classification and selection using fuzzy‐ant colony optimisation approach , 2015, Softw. Pract. Exp..

[35]  Wei Sun,et al.  Multi-objective test case prioritization for GUI applications , 2013, SAC '13.

[36]  Robert M. Hierons,et al.  Multi-objective optimisation for regression testing , 2016, Inf. Sci..

[37]  Antonia Bertolino,et al.  An assessment of operational coverage as both an adequacy and a selection criterion for operational profile based testing , 2017, Software Quality Journal.

[38]  Boris Beizer,et al.  Software testing techniques (2. ed.) , 1990 .

[39]  Bestoun S. Ahmed,et al.  Achievement of minimized combinatorial test suite for configuration-aware software functional testing using the Cuckoo Search algorithm , 2015, Inf. Softw. Technol..

[40]  Bharti Suri,et al.  Analyzing Test Case Selection using Proposed Hybrid Technique based on BCO and Genetic Algorithm and a Comparison with ACO , 2012 .

[41]  Raffaela Mirandola,et al.  A Hybrid Approach for Multi-attribute QoS Optimisation in Component Based Software Systems , 2010, QoSA.

[42]  Amir Saman Kheirkhah,et al.  Two hybrid tabu search–firefly algorithms for the capacitated job shop scheduling problem with sequence-dependent setup cost , 2015, Int. J. Comput. Integr. Manuf..

[43]  B. Eswara Reddy,et al.  DDF: Diversity Dragonfly Algorithm for cost-aware test suite minimization approach for software testing , 2017, 2017 International Conference on Intelligent Computing and Control Systems (ICICCS).

[44]  Gordon Fraser,et al.  Behaviourally Adequate Software Testing , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[45]  Praveen Ranjan Srivastava,et al.  Application of genetic algorithm and tabu search in software testing , 2011, Bangalore Compute Conf..

[46]  Mark Harman,et al.  Optimizing for the Number of Tests Generated in Search Based Test Data Generation with an Application to the Oracle Cost Problem , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[47]  Arvinder Kaur,et al.  A Comprehensive Comparison of Ant Colony and Hybrid Particle Swarm Optimization Algorithms Through Test Case Selection , 2018 .

[48]  Manoj Kumar,et al.  Optimization of Test Cases using Soft Computing Techniques : A Critical Review , 2012 .

[49]  A. A. Ahmed,et al.  Software testing suite prioritization using multi-criteria fitness function , 2012, 2012 22nd International Conference on Computer Theory and Applications (ICCTA).

[50]  Antonia Bertolino,et al.  Software Testing Research: Achievements, Challenges, Dreams , 2007, Future of Software Engineering (FOSE '07).

[51]  Mark Harman,et al.  Empirical evaluation of pareto efficient multi-objective regression test case prioritisation , 2015, ISSTA.

[52]  Sanjay Kumar,et al.  Automated Test Data Generation Using Cuckoo Search and Tabu Search (CSTS) Algorithm , 2012, J. Intell. Syst..

[53]  Gregory Gay,et al.  The Risks of Coverage-Directed Test Case Generation , 2015, IEEE Transactions on Software Engineering.

[54]  Ricardo B. C. Prudêncio,et al.  A Hybrid Binary Multi-objective Particle Swarm Optimization with Local Search for Test Case Selection , 2014, 2014 Brazilian Conference on Intelligent Systems.

[55]  Xin-She Yang,et al.  Bat algorithm for multi-objective optimisation , 2011, Int. J. Bio Inspired Comput..

[56]  Radziah Mohamad,et al.  Systematic Mapping Study in Automatic Test Case Generation , 2014, SoMeT.

[57]  G. Radhamani,et al.  Fuzzy C Means (FCM) Clustering Based Hybrid Swarm Intelligence Algorithm for Test Case Optimization , 2014 .

[58]  GORDON FRASER,et al.  A Large-Scale Evaluation of Automated Unit Test Generation Using EvoSuite , 2014, ACM Trans. Softw. Eng. Methodol..

[59]  Mark Harman,et al.  Using hybrid algorithm for Pareto efficient multi-objective test suite minimisation , 2010, J. Syst. Softw..

[60]  Ali Ahsan,et al.  Exploration and analysis of regression test suite optimization , 2014, SOEN.

[61]  Lionel C. Briand,et al.  Coverage‐based regression test case selection, minimization and prioritization: a case study on an industrial system , 2015, Softw. Test. Verification Reliab..

[62]  Giuseppe Scanniello,et al.  A Multi-Objective Technique to Prioritize Test Cases Based on Latent Semantic Indexing , 2012, 2012 16th European Conference on Software Maintenance and Reengineering.

[63]  Gregory Gay,et al.  Using Search-Based Test Generation to Discover Real Faults in Guava , 2017, SSBSE.

[64]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[65]  Hadi Hemmati,et al.  How Effective Are Code Coverage Criteria? , 2015, 2015 IEEE International Conference on Software Quality, Reliability and Security.

[66]  Neil Walkinshaw,et al.  Assessing Test Adequacy for Black-Box Systems without Specifications , 2011, ICTSS.

[67]  Mark Harman,et al.  Pareto efficient multi-objective test case selection , 2007, ISSTA '07.

[68]  Shuai Wang,et al.  Enhancing Test Case Prioritization in an Industrial Setting with Resource Awareness and Multi-objective Search , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C).

[69]  Gordon Fraser,et al.  Whole Test Suite Generation , 2013, IEEE Transactions on Software Engineering.

[70]  Farhad Soleimanian Gharehchopogh,et al.  A Hybrid of Ant Colony Optimization and Chaos Optimization Algorithms Approach for Software Cost Estimation , 2015 .

[71]  Márcio Eduardo Delamaro,et al.  Test Case Selection: A Systematic Literature Review , 2014, Int. J. Softw. Eng. Knowl. Eng..

[72]  Andrea De Lucia,et al.  On the role of diversity measures for multi-objective test case selection , 2012, 2012 7th International Workshop on Automation of Software Test (AST).

[73]  Shin Yoo,et al.  A Theoretical and Empirical Study of Diversity-Aware Mutation Adequacy Criterion , 2018, IEEE Transactions on Software Engineering.