Enhancing Efficiency of the Test Case Prioritization Technique by Improving the Rate of Fault Detection

Test case prioritization techniques organize test cases for implementation in a manner that enhance their efficacy in accordance with some performance goal. The main aim of regression testing is to test the amended software to assure that the amendments performed in software are correct. It is not always feasible to retest entire test cases in a test suite due to limited resources. Therefore, it is necessary to develop some effective techniques that can enhance the regression testing effectiveness by organizing the test cases in an order following some testing criterion. One possible criterion of such prioritization is to enhance a test suite’s fault detection rate. It aspires to arrange test cases in an order that higher priority test cases run earlier than lower ones. This paper proposed a methodology for prioritizing regression test cases based on four factors namely the rate of fault detection, the number of faults detected, the test case ability of risk detection and the test case effectiveness. The proposed approach is implemented on two projects. The resultant test case order is analyzed with other prioritization techniques such as no prioritization, random prioritization, reverse prioritization, optimal prioritization and along with previous works for project 1. We have applied our proposed approach for prioritizing test cases in an order that maximize fault coverage with least test suite execution and compared its effectiveness with other orderings. The result of proposed approach shows higher average percentage of fault detected value and outperforms all other approaches.

[1]  Chayanika Sharma,et al.  A genetic algorithm based approach for prioritization of test case scenarios in static testing , 2011, 2011 2nd International Conference on Computer and Communication Technology (ICCCT-2011).

[2]  Gregg Rothermel,et al.  A safe, efficient regression test selection technique , 1997, TSEM.

[3]  Mary Lou Soffa,et al.  A methodology for controlling the size of a test suite , 1993, TSEM.

[4]  Roger S. Pressman,et al.  Software Engineering: A Practitioner's Approach , 1982 .

[5]  Arvinder Kaur,et al.  Test case prioritization using ant colony optimization , 2010, SOEN.

[6]  Gregg Rothermel,et al.  Cost-cognizant Test Case Prioritization , 2006 .

[7]  Phyllis G. Frankl,et al.  Pythia: a regression test selection tool based on textual differencing , 1997 .

[8]  Neelam Gupta,et al.  Test suite reduction with selective redundancy , 2005, 21st IEEE International Conference on Software Maintenance (ICSM'05).

[9]  Gregg Rothermel,et al.  Test case prioritization: an empirical study , 1999, Proceedings IEEE International Conference on Software Maintenance - 1999 (ICSM'99). 'Software Maintenance for Business Change' (Cat. No.99CB36360).

[10]  Durga Prasad Mohapatra,et al.  Multi-objective test prioritization via a genetic algorithm , 2014, Innovations in Systems and Software Engineering.

[11]  Laurie A. Williams,et al.  Prioritization of Regression Tests using Singular Value Decomposition with Empirical Change Records , 2007, The 18th IEEE International Symposium on Software Reliability (ISSRE '07).

[12]  Boris Beizer,et al.  Software System Testing and Quality Assurance , 1984 .

[13]  Neelam Gupta,et al.  Test Case Prioritization Using Relevant Slices , 2006, 30th Annual International Computer Software and Applications Conference (COMPSAC'06).

[14]  Frank Tip,et al.  Chianti: a tool for change impact analysis of java programs , 2004, OOPSLA.

[15]  W. Eric Wong,et al.  Effect of test set minimization on fault detection effectiveness , 1998 .

[16]  David Leon,et al.  Finding failures by cluster analysis of execution profiles , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[17]  D. Jeya Mala,et al.  A non-pheromone based intelligent swarm optimization technique in software test suite optimization , 2009, 2009 International Conference on Intelligent Agent & Multi-Agent Systems.

[18]  Richard G. Hamlet,et al.  Probable Correctness Theory , 1987, Inf. Process. Lett..

[19]  Gregg Rothermel,et al.  Analyzing Regression Test Selection Techniques , 1996, IEEE Trans. Software Eng..

[20]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[21]  Thomas Ball,et al.  On the limit of control flow analysis for regression test selection , 1998, ISSTA '98.

[22]  Bogdan Korel,et al.  Model-based test prioritization heuristic methods and their evaluation , 2007, A-MOST '07.

[23]  Bo Qu,et al.  Test Case Prioritization for Black Box Testing , 2007, 31st Annual International Computer Software and Applications Conference (COMPSAC 2007).

[24]  Elaine J. Weyuker,et al.  The Automatic Generation of Load Test Suites and the Assessment of the Resulting Software , 1995, IEEE Trans. Software Eng..

[25]  Antonia Bertolino,et al.  Using Spanning Sets for Coverage Testing , 2003, IEEE Trans. Software Eng..

[26]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[27]  David S. Rosenblum,et al.  TESTTUBE: a system for selective regression testing , 1994, Proceedings of 16th International Conference on Software Engineering.

[28]  R. Kavitha,et al.  Requirement based test case prioritization , 2010, 2010 INTERNATIONAL CONFERENCE ON COMMUNICATION CONTROL AND COMPUTING TECHNOLOGIES.

[29]  Izzat Alsmadi,et al.  Test Cases Reduction and Selection Optimization in Testing Web Services , 2012 .

[30]  Chin-Yu Huang,et al.  Design and Analysis of Cost-Cognizant Test Case Prioritization Using Genetic Algorithm with Test History , 2010, 2010 IEEE 34th Annual Computer Software and Applications Conference.

[31]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[32]  Mark Harman,et al.  Search Algorithms for Regression Test Case Prioritization , 2007, IEEE Transactions on Software Engineering.

[33]  Gregg Rothermel,et al.  An empirical study of the effects of minimization on the fault detection capabilities of test suites , 1998, Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272).

[34]  Gregg Rothermel,et al.  Selecting a Cost-Effective Test Case Prioritization Technique , 2004, Software Quality Journal.

[35]  Joseph Robert Horgan,et al.  Test set size minimization and fault detection effectiveness: a case study in a space application , 1997, Proceedings Twenty-First Annual International Computer Software and Applications Conference (COMPSAC'97).

[36]  Ricardo B. C. Prudêncio,et al.  A Constrained Particle Swarm Optimization Approach for Test Case Selection , 2010, SEKE.

[37]  Gregg Rothermel,et al.  Incorporating varying test costs and fault severities into test case prioritization , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[38]  Mary Lou Soffa,et al.  TimeAware test suite prioritization , 2006, ISSTA '06.

[39]  Laurie A. Williams,et al.  Towards the prioritization of system test cases , 2014, Softw. Test. Verification Reliab..

[40]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[41]  Manika Tyagi,et al.  An Approach for Test Case Prioritization Based on Three Factors , 2015 .

[42]  Md. Imrul Kayes Test case prioritization for regression testing based on fault dependency , 2011, 2011 3rd International Conference on Electronics Computer Technology.

[43]  K. K. Aggarwal,et al.  Software reuse metrics for object-oriented systems , 2005, Third ACIS Int'l Conference on Software Engineering Research, Management and Applications (SERA'05).

[44]  David Leon,et al.  An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows , 2007, IEEE Transactions on Software Engineering.

[45]  Adam A. Porter,et al.  A history-based test prioritization technique for regression testing in resource constrained environments , 2002, ICSE '02.

[46]  Jong Sou Park,et al.  Applying Particle Swarm Optimization to Prioritizing Test Cases for Embedded Real Time Software Retesting , 2008, 2008 IEEE 8th International Conference on Computer and Information Technology Workshops.

[47]  R. Uma Maheswari,et al.  A novel approach for test case prioritization , 2013, 2013 IEEE International Conference on Computational Intelligence and Computing Research.

[48]  Gregg Rothermel,et al.  Test Case Prioritization: A Family of Empirical Studies , 2002, IEEE Trans. Software Eng..

[49]  Yuri Chernak,et al.  Validating and Improving Test-Case Effectiveness , 2001, IEEE Softw..

[50]  Richard G. Hamlet,et al.  Testing Programs with the Aid of a Compiler , 1977, IEEE Transactions on Software Engineering.

[51]  Joseph Robert Horgan,et al.  A study of effective regression testing in practice , 1997, Proceedings The Eighth International Symposium on Software Reliability Engineering.