sOrTES: A Supportive Tool for Stochastic Scheduling of Manual Integration Test Cases

The main goal of software testing is to detect as many hidden bugs as possible in the final software product before release. Generally, a software product is tested by executing a set of test cases, which can be performed manually or automatically. The number of test cases which are required to test a software product depends on several parameters such as the product type, size, and complexity. Executing all test cases with no particular order can lead to waste of time and resources. Test optimization can provide a partial solution for saving time and resources which can lead to the final software product being released earlier. In this regard, test case selection, prioritization, and scheduling can be considered as possible solutions for test optimization. Most of the companies do not provide direct support for ranking test cases on their own servers. In this paper, we introduce, apply, and evaluate sOrTES as our decision support system for manual integration of test scheduling. sOrTES is a Python-based supportive tool which schedules manual integration test cases which are written in a natural language text. The feasibility of sOrTES is studied by an empirical evaluation which has been performed on a railway use-case at Bombardier Transportation, Sweden. The empirical evaluation indicates that around 40 % of testing failure can be avoided by using the proposed execution schedules by sOrTES, which leads to an increase in the requirements coverage of up to 9.6%.

[1]  Colin Robson,et al.  Real world research : a resource for users of social research methods in applied settings , 2011 .

[2]  Xiaolin Wang,et al.  History-based dynamic test case prioritization for requirement properties in regression testing , 2016, CSED@ICSE.

[3]  Per Runeson,et al.  Improving Regression Testing Transparency and Efficiency with History-Based Prioritization -- An Industrial Case Study , 2011, 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation.

[4]  Mehrdad Saadatmand,et al.  Multi-Criteria Test Case Prioritization Using Fuzzy Analytic Hierarchy Process , 2015, ICSEA 2015.

[5]  J.M. Schopf,et al.  Stochastic Scheduling , 1999, ACM/IEEE SC 1999 Conference (SC'99).

[6]  Reinhard German,et al.  Dependency-Based Test Case Selection and Prioritization in Embedded Systems , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[7]  Gregg Rothermel,et al.  Incorporating varying test costs and fault severities into test case prioritization , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[8]  Tao Xie,et al.  Time-aware test-case prioritization using integer linear programming , 2009, ISSTA.

[9]  Mika Mäntylä,et al.  Are test cases needed? Replicated comparison between exploratory and test-case-based software testing , 2014, Empirical Software Engineering.

[10]  Andreas Podelski,et al.  If A Fails, Can B Still Succeed? Inferring Dependencies between Test Results in Automotive System Testing , 2015, 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST).

[11]  Daniel Sundmark,et al.  Cost-Benefit Analysis of Using Dependency Knowledge at Integration Testing , 2016, PROFES.

[12]  Cleotilde Gonzalez,et al.  Instance-based learning in dynamic decision making , 2003, Cogn. Sci..

[13]  Wasif Afzal,et al.  Functional Dependency Detection for Integration Test Cases , 2018, 2018 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C).

[14]  Daniel Sundmark,et al.  Dynamic Integration Test Selection Based on Test Case Dependencies , 2016, 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[15]  Mary Lou Soffa,et al.  TimeAware test suite prioritization , 2006, ISSTA '06.

[16]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[17]  Adam A. Porter,et al.  A history-based test prioritization technique for regression testing in resource constrained environments , 2002, ICSE '02.

[18]  Gregg Rothermel,et al.  Test Case Prioritization: A Family of Empirical Studies , 2002, IEEE Trans. Software Eng..

[19]  Per Runeson,et al.  Guidelines for conducting and reporting case study research in software engineering , 2009, Empirical Software Engineering.

[20]  Shuai Wang,et al.  Enhancing Test Case Prioritization in an Industrial Setting with Resource Awareness and Multi-objective Search , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C).

[21]  Mark Harman,et al.  Search Algorithms for Regression Test Case Prioritization , 2007, IEEE Transactions on Software Engineering.

[22]  Tim Miller,et al.  Using Dependency Structures for Prioritization of Functional Test Suites , 2013, IEEE Transactions on Software Engineering.

[23]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[24]  Laurie A. Williams,et al.  System test case prioritization of new and regression test cases , 2005, 2005 International Symposium on Empirical Software Engineering, 2005..

[25]  Gaurav Kumar,et al.  Software testing optimization through test suite reduction using fuzzy clustering , 2013, CSI Transactions on ICT.

[26]  Wasif Afzal,et al.  ESPRET: A tool for execution time estimation of manual test cases , 2018, J. Syst. Softw..

[27]  Elaine J. Weyuker,et al.  Experience Report: Automated System Level Regression Test Prioritization Using Multiple Factors , 2018 .

[28]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[29]  Michael Felderer,et al.  Cluster-Based Test Scheduling Strategies Using Semantic Relationships between Test Specifications , 2018, 2018 IEEE/ACM 5th International Workshop on Requirements Engineering and Testing (RET).

[30]  Rudolf Ramler,et al.  An empirical study on the application of mutation testing for a safety-critical industrial software system , 2017, SAC.

[31]  Hadi Hemmati,et al.  A similarity-based approach for test case prioritization using historical failure data , 2015, 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE).

[32]  José Niño-Mora.  Stochastic Scheduling , 2009, Encyclopedia of Optimization.

[33]  Eunseok Lee,et al.  History-Based Test Case Prioritization for Failure Information , 2016, 2016 23rd Asia-Pacific Software Engineering Conference (APSEC).

[34]  Julius Beneoluchi Odili,et al.  Solving the Traveling Salesman's Problem Using the African Buffalo Optimization , 2016, Comput. Intell. Neurosci..

[35]  Amitabh Srivastava,et al.  Effectively prioritizing tests in development environment , 2002, ISSTA '02.

[36]  Mouaaz Nahas,et al.  Developing Scheduler Test Cases to Verify Scheduler Implementations In Time-Triggered Embedded Systems , 2016 .

[37]  Joseph Robert Horgan,et al.  A study of effective regression testing in practice , 1997, Proceedings The Eighth International Symposium on Software Reliability Engineering.

[38]  Sue J. Waite,et al.  Real world research: a resource for users of social research methods in applied settings (3rd ed.) , 2012 .

[39]  Mark Harman.  Making the Case for MORTO: Multi Objective Regression Test Optimization , 2011, 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.

[40]  Daniel Sundmark,et al.  A Comparative Study of Manual and Automated Testing for Industrial Control Software , 2017, 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST).

[41]  Daniel Sundmark,et al.  Towards Earlier Fault Detection by Value-Driven Prioritization of Test Cases Using Fuzzy TOPSIS , 2016, ITNG 2016.