Test case prioritization: a systematic mapping study

Test case prioritization techniques, which are used to improve the cost-effectiveness of regression testing, order test cases in such a way that those cases that are expected to outperform others in detecting software faults are run earlier in the testing phase. The objective of this study is to examine what kind of techniques have been widely used in papers on this subject, determine which aspects of test case prioritization have been studied, provide a basis for the improvement of test case prioritization research, and evaluate the current trends of this research area. We searched for papers in the following five electronic databases: IEEE Explorer, ACM Digital Library, Science Direct, Springer, and Wiley. Initially, the search string retrieved 202 studies, but upon further examination of titles and abstracts, 120 papers were identified as related to test case prioritization. There exists a large variety of prioritization techniques in the literature, with coverage-based prioritization techniques (i.e., prioritization in terms of the number of statements, basic blocks, or methods test cases cover) dominating the field. The proportion of papers on model-based techniques is on the rise, yet the growth rate is still slow. The proportion of papers that use datasets from industrial projects is found to be 64 %, while those that utilize public datasets for validation are only 38 %. On the basis of this study, the following recommendations are provided for researchers: (1) Give preference to public datasets rather than proprietary datasets; (2) develop more model-based prioritization methods; (3) conduct more studies on the comparison of prioritization methods; (4) always evaluate the effectiveness of the proposed technique with well-known evaluation metrics and compare the performance with the existing methods; (5) publish surveys and systematic review papers on test case prioritization; and (6) use datasets from industrial projects that represent real industrial problems.

[1]  Philip Samuel,et al.  Web services regression test case prioritization , 2010, 2010 International Conference on Computer Information Systems and Industrial Management Applications (CISIM).

[2]  Mary Lou Soffa,et al.  Test suite reduction and prioritization with call trees , 2007, ASE '07.

[3]  K. Ramasamy,et al.  Incorporating varying requirement priorities and costs in test case prioritization for new and regression testing , 2008, 2008 International Conference on Computing, Communication and Networking.

[4]  Per Runeson,et al.  A systematic review on regression test selection techniques , 2010, Inf. Softw. Technol..

[5]  Chin-Yu Huang,et al.  Design and Analysis of Cost-Cognizant Test Case Prioritization Using Genetic Algorithm with Test History , 2010, 2010 IEEE 34th Annual Computer Software and Applications Conference.

[6]  Bogdan Korel,et al.  Model-based test prioritization heuristic methods and their evaluation , 2007, A-MOST '07.

[7]  Ladan Tahvildari,et al.  A Prioritization Approach for Software Test Cases Based on Bayesian Networks , 2007, FASE.

[8]  Klaus Pohl,et al.  An automated technique for risk-based test case generation and prioritization , 2008, AST '08.

[9]  Adam A. Porter,et al.  A history-based test prioritization technique for regression testing in resource constrained environments , 2002, ICSE '02.

[10]  Bo Qu,et al.  Test Case Prioritization for Black Box Testing , 2007, 31st Annual International Computer Software and Applications Conference (COMPSAC 2007).

[11]  Bo Qu,et al.  Test Case Prioritization Based on Varying Testing Requirement Priorities and Test Case Costs , 2007, Seventh International Conference on Quality Software (QSIC 2007).

[12]  Sejun Kim,et al.  An effective fault aware test case prioritization by incorporating a fault localization technique , 2010, ESEM '10.

[13]  Gregg Rothermel,et al.  Test Case Prioritization: A Family of Empirical Studies , 2002, IEEE Trans. Software Eng..

[14]  Tai-Hoon Kim,et al.  Test Case Minimization and Prioritization Using CMIMX Technique , 2009, FGIT-ASEA.

[15]  Neelam Gupta,et al.  Experiments with test case prioritization using relevant slices , 2008, J. Syst. Softw..

[16]  Orest Pilskalns,et al.  Regression testing with UML software designs: A survey , 2009, J. Softw. Maintenance Res. Pract..

[17]  W. K. Chan,et al.  Point-of-Interest Aware Test Case Prioritization: Methods and Experiments , 2010, 2010 10th International Conference on Quality Software.

[18]  Pearl Brereton,et al.  Systematic literature reviews in software engineering - A systematic literature review , 2009, Inf. Softw. Technol..

[19]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[20]  Jun Huang,et al.  Regression Test Generation Approach Based on Tree-Structured Analysis , 2010, 2010 International Conference on Computational Science and Its Applications.

[21]  Gregory M. Kapfhammer.  Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward , 2011, 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.

[22]  Mary Jean Harrold,et al.  Regression test selection on system requirements , 2008, ISEC '08.

[23]  Bo Jiang,et al.  On the Integration of Test Adequacy, Test Case Prioritization, and Statistical Fault Localization , 2010, 2010 10th International Conference on Quality Software.

[24]  Tore Dybå,et al.  Evidence-based software engineering , 2004, Proceedings. 26th International Conference on Software Engineering.

[25]  Fevzi Belli,et al.  Prioritizing Coverage-Oriented Testing Process - An Adaptive-Learning-Based Approach and Case Study , 2007, 31st Annual International Computer Software and Applications Conference (COMPSAC 2007).

[26]  Hyunsook Do,et al.  The Effectiveness of Regression Testing Techniques in Reducing the Occurrence of Residual Defects , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[27]  Y. Fazlalizadeh,et al.  Prioritizing test cases for resource constraint environments using historical test case performance data , 2009, 2009 2nd IEEE International Conference on Computer Science and Information Technology.

[28]  Atif M. Memon,et al.  Developing a Single Model and Test Prioritization Strategies for Event-Driven Software , 2011, IEEE Transactions on Software Engineering.

[29]  Gregory M. Kapfhammer,et al.  An empirical study of incorporating cost into test suite reduction and prioritization , 2009, SAC '09.

[30]  Hrushikesha Mohanty,et al.  Prioritizing Use Cases to Aid Ordering of Scenarios , 2009, 2009 Third UKSim European Symposium on Computer Modeling and Simulation.

[31]  T. H. Tse,et al.  Test case prioritization for regression testing of service-oriented business applications , 2009, WWW '09.

[32]  Gregg Rothermel,et al.  An empirical study of the effect of time constraints on the cost-benefits of regression testing , 2008, SIGSOFT '08/FSE-16.

[33]  Dianxiang Xu,et al.  Prioritizing State-Based Aspect Tests , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[34]  Myra B. Cohen,et al.  Combinatorial Interaction Regression Testing: A Study of Test Case Generation and Prioritization , 2007, 2007 IEEE International Conference on Software Maintenance.

[35]  Hong Mei,et al.  Jtop: Managing JUnit Test Cases in Absence of Coverage Information , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[36]  T. H. Tse,et al.  Taking Advantage of Service Selection: A Study on the Testing of Location-Based Web Services Through Test Case Prioritization , 2010, 2010 IEEE International Conference on Web Services.

[37]  Daoxu Chen,et al.  A Hybrid Approach to Build Prioritized Pairwise Interaction Test Suites , 2009, 2009 International Conference on Computational Intelligence and Software Engineering.

[38]  Baowen Xu,et al.  Test Case Prioritization for Multiple Processing Queues , 2008, 2008 International Symposium on Information Science and Engineering.

[39]  Per Runeson,et al.  Empirical evaluations of regression test selection techniques: a systematic review , 2008, ESEM '08.

[40]  Magne Jørgensen,et al.  A Systematic Review of Software Development Cost Estimation Studies , 2007, IEEE Transactions on Software Engineering.

[41]  Zhi Quan Zhou.  Using Coverage Information to Guide Test Case Selection in Adaptive Random Testing , 2010, 2010 IEEE 34th Annual Computer Software and Applications Conference Workshops.

[42]  Abraham Kandel,et al.  Effective Black-Box Testing with Genetic Algorithms , 2005, Haifa Verification Conference.

[43]  Gregory M. Kapfhammer,et al.  Using synthetic test suites to empirically compare search-based and greedy prioritizers , 2010, GECCO '10.

[44]  S. A. Sahaaya Arul Mary,et al.  Factor oriented requirement coverage based system test case prioritization of new and regression test cases , 2009, Inf. Softw. Technol..

[45]  Gary King,et al.  An Introduction to the Dataverse Network as an Infrastructure for Data Sharing , 2007 .

[46]  Laurie A. Williams,et al.  System test case prioritization of new and regression test cases , 2005, 2005 International Symposium on Empirical Software Engineering, 2005..

[47]  Nashat Mansour,et al.  Regression Test Selection for C# Programs , 2009, Adv. Softw. Eng..

[48]  Bogdan Korel,et al.  Experimental Comparison of Code-Based and Model-Based Test Prioritization , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[49]  Hrushikesha Mohanty,et al.  Prioritization of Scenarios Based on UML Activity Diagrams , 2009, 2009 First International Conference on Computational Intelligence, Communication Systems and Networks.

[50]  Gregg Rothermel,et al.  Incorporating varying test costs and fault severities into test case prioritization , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[51]  Neelam Gupta,et al.  Test Case Prioritization Using Relevant Slices , 2006, 30th Annual International Computer Software and Applications Conference (COMPSAC'06).

[52]  Banu Diri,et al.  A systematic review of software fault prediction studies , 2009, Expert Syst. Appl..

[53]  Charles J. Colbourn,et al.  Constructing interaction test suites with greedy algorithms , 2005, ASE '05.

[54]  Pearl Brereton,et al.  The value of mapping studies - A participant-observer case study , 2010, EASE.

[55]  Tsong Yueh Chen,et al.  How Well Do Test Case Prioritization Techniques Support Statistical Fault Localization , 2009, 2009 33rd Annual IEEE International Computer Software and Applications Conference.

[56]  Suresh Jagannathan,et al.  PHALANX: a graph-theoretic framework for test case prioritization , 2008, SAC '08.

[57]  Akif Günes Koru,et al.  Prioritizing User-Session-Based Test Cases for Web Applications Testing , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[58]  N. Ramaraj,et al.  Enhancing the Efficiency of Regression Testing through Intelligent Agents , 2007, International Conference on Computational Intelligence and Multimedia Applications (ICCIMA 2007).

[59]  Mary Lou Soffa,et al.  TimeAware test suite prioritization , 2006, ISSTA '06.

[60]  Mary Lou Soffa,et al.  Efficient time-aware prioritization with knapsack solvers , 2007, WEASELTech '07.

[61]  Ladan Tahvildari,et al.  An Empirical Study on Bayesian Network-based Approach for Test Case Prioritization , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[62]  Chin-Yu Huang,et al.  Design and analysis of GUI test-case prioritization using weight-based methods , 2010, J. Syst. Softw..

[63]  Gregg Rothermel,et al.  Understanding the effects of changes on the cost‐effectiveness of regression testing techniques , 2003, Softw. Test. Verification Reliab..

[64]  Mary Jean Harrold,et al.  Test-Suite Reduction and Prioritization for Modified Condition/Decision Coverage , 2003, IEEE Trans. Software Eng..

[65]  Saeed Parsa,et al.  Incorporating Historical Test Case Performance Data and Resource Constraints into Test Case Prioritization , 2009, TAP@TOOLS.

[66]  Rajesh Subramanyan,et al.  Supporting Concern-Based Regression Testing and Prioritization in a Model-Driven Environment , 2010, 2010 IEEE 34th Annual Computer Software and Applications Conference Workshops.

[67]  Baowen Xu,et al.  Test Case Prioritization for Web Service Regression Testing , 2010, 2010 Fifth IEEE International Symposium on Service Oriented System Engineering.

[68]  Gregg Rothermel,et al.  Understanding and measuring the sources of variation in the prioritization of regression test suites , 2001, Proceedings Seventh International Software Metrics Symposium.

[69]  Magne Jørgensen,et al.  A Systematic Review of Software Development Cost Estimation Studies , 2007 .

[70]  Atif M. Memon,et al.  Test suite prioritization by interaction coverage , 2007, DOSTA '07.

[71]  Jie Zhu,et al.  Prioritized Test Generation Strategy for Pair-Wise Testing , 2009, 2009 15th IEEE Pacific Rim International Symposium on Dependable Computing.

[72]  Gregg Rothermel,et al.  On test suite composition and cost-effective regression testing , 2004, TSEM.

[73]  Mary Lou Soffa,et al.  Using coverage effectiveness to evaluate test suite prioritizations , 2007, WEASELTech '07.

[74]  Chayanika Sharma,et al.  Prioritization of test case scenarios derived from activity diagram using genetic algorithm , 2010, 2010 International Conference on Computer and Communication Technology (ICCCT).

[75]  Ysong Yueh Yu,et al.  Generating, selecting and prioritizing test cases from specifications with tool support , 2003, Third International Conference on Quality Software, 2003. Proceedings..

[76]  Yuen-Tak Yu,et al.  Prioritization of Test Cases in MUMCUT Test Sets: An Empirical Study , 2002, Ada-Europe.

[77]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[78]  Marek Grzegorz Stochel,et al.  Testing Optimization for Mission-Critical, Complex, Distributed Systems , 2008, 2008 32nd Annual IEEE International Computer Software and Applications Conference.

[79]  Ossi Taipale,et al.  Test case selection and prioritization: risk-based or design-based? , 2010, ESEM '10.

[80]  Kai Petersen,et al.  Systematic Mapping Studies in Software Engineering , 2008, EASE.

[81]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[82]  Gregg Rothermel,et al.  Prioritizing JUnit Test Cases: An Empirical Assessment and Cost-Benefits Analysis , 2006, Empirical Software Engineering.

[83]  Mark Harman,et al.  Search Algorithms for Regression Test Case Prioritization , 2007, IEEE Transactions on Software Engineering.

[84]  Fevzi Belli,et al.  Model-Based Test Prioritizing - A Comparative Soft-Computing Approach and Case Studies , 2009, KI.

[85]  Gregg Rothermel,et al.  Selecting a Cost-Effective Test Case Prioritization Technique , 2004, Software Quality Journal.

[86]  Jianjun Yuan,et al.  Constructing Prioritized Interaction Test Suite with Interaction Relationship , 2010, 2010 Second International Workshop on Education Technology and Computer Science.

[87]  Gregg Rothermel,et al.  On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques , 2006, IEEE Transactions on Software Engineering.

[88]  Xiaoqing Bai,et al.  An approach to generate the thin-threads from the UML diagrams , 2004, Proceedings of the 28th Annual International Computer Software and Applications Conference, 2004. COMPSAC 2004..

[89]  Hyuncheol Park,et al.  Historical Value-Based Approach for Cost-Cognizant Test Case Prioritization to Improve the Effectiveness of Regression Testing , 2008, 2008 Second International Conference on Secure System Integration and Reliability Improvement.

[90]  Gregg Rothermel,et al.  Empirical studies of test case prioritization in a JUnit testing environment , 2004, 15th International Symposium on Software Reliability Engineering.

[91]  T. H. Tse,et al.  Tag-Based Techniques for Black-Box Test Case Prioritization for Service Testing , 2009, 2009 Ninth International Conference on Quality Software.

[92]  Ahmed E. Hassan,et al.  Prioritizing Unit Test Creation for Test-Driven Maintenance of Legacy Systems , 2010, 2010 10th International Conference on Quality Software.

[93]  Fevzi Belli,et al.  Coverage-Oriented, Prioritized Testing - A Fuzzy Clustering Approach and Case Study , 2007, LADC.

[94]  Augusto Sampaio,et al.  Test case prioritization based on data reuse an experimental study , 2009, 2009 3rd International Symposium on Empirical Software Engineering and Measurement.

[95]  Tao Xie,et al.  Quota-constrained test-case prioritization for regression testing of service-centric systems , 2008, 2008 IEEE International Conference on Software Maintenance.

[96]  Gregg Rothermel,et al.  Putting Your Best Tests Forward , 2003, IEEE Softw..

[97]  Arjan J. C. van Gemund,et al.  Prioritizing Tests for Software Fault Localization , 2010, 2010 10th International Conference on Quality Software.

[98]  Gregory M. Kapfhammer,et al.  Towards the prioritization of regression test suites with data flow information , 2005, SAC '05.

[99]  Emerson Cabrera Paraiso,et al.  Regression test cases prioritization using Failure Pursuit Sampling , 2010, 2010 10th International Conference on Intelligent Systems Design and Applications.

[100]  Paolo Tonella,et al.  Using the Case-Based Ranking Methodology for Test Case Prioritization , 2006, 2006 22nd IEEE International Conference on Software Maintenance.

[101]  Gregg Rothermel,et al.  Modeling the cost-benefits tradeoffs for regression testing techniques , 2002, International Conference on Software Maintenance, 2002. Proceedings..

[102]  Laurie A. Williams,et al.  Prioritization of Regression Tests using Singular Value Decomposition with Empirical Change Records , 2007, The 18th IEEE International Symposium on Software Reliability (ISSRE '07).

[103]  P.R. Srivastava.  Model for optimizing software testing period using non homogenous poisson process based on cumulative test case prioritization , 2008, TENCON 2008 - 2008 IEEE Region 10 Conference.

[104]  T. H. Tse,et al.  Adaptive Random Test Case Prioritization , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[105]  Saif Ur Rehman Malik,et al.  The impact of test case reduction and prioritization on software testing effectiveness , 2009, 2009 International Conference on Emerging Technologies.

[106]  Pearl Brereton,et al.  Performing systematic literature reviews in software engineering , 2006, ICSE.

[107]  Gregg Rothermel,et al.  The Effects of Time Constraints on Test Case Prioritization: A Series of Controlled Experiments , 2010, IEEE Transactions on Software Engineering.

[108]  Tao Xie,et al.  Time-aware test-case prioritization using integer linear programming , 2009, ISSTA.

[109]  Jianjun Zhao,et al.  Test Case Prioritization Based on Analysis of Program Structure , 2008, 2008 15th Asia-Pacific Software Engineering Conference.

[110]  Hema Srikanth,et al.  On the Economics of Requirements-Based Test Case Prioritization , 2005 .

[111]  Gregory M. Kapfhammer,et al.  Empirically studying the role of selection operators duringsearch-based test suite prioritization , 2010, GECCO '10.

[112]  Myra B. Cohen,et al.  Reducing Field Failures in System Configurable Software: Cost-Based Prioritization , 2009, 2009 20th International Symposium on Software Reliability Engineering.

[113]  Farn Wang,et al.  Test Plan Generation for Concurrent Real-Time Systems Based on Zone Coverage Analysis , 2008, TestCom/FATES.

[114]  T. H. Tse,et al.  XML-manipulating test case prioritization for XML-manipulating services , 2011, J. Syst. Softw..

[115]  R. Kavitha,et al.  Requirement based test case prioritization , 2010, 2010 INTERNATIONAL CONFERENCE ON COMMUNICATION CONTROL AND COMPUTING TECHNOLOGIES.

[116]  Lu Zhang,et al.  Prioritizing JUnit test cases in absence of coverage information , 2009, 2009 IEEE International Conference on Software Maintenance.

[117]  Mark Harman,et al.  Test prioritization using system models , 2005, 21st IEEE International Conference on Software Maintenance (ICSM'05).