State-based models in regression test suite prioritization

Testing software products is very expensive and time consuming, especially for large software systems with extensive regression testing. During regression testing, a modified system is often re-tested using an existing test suite. Since test suites can be very large, testers are interested in detecting faults in the modified system as early as possible. Test prioritization tries to order test cases for execution in a way that increases the chances of the early detection of faults. Most of the existing test prioritization methods are based on the code of the system under test, but model-based test prioritization has been lately proposed. Most of the existing model-based test prioritization methods can be used only when models are modified during system maintenance. In this paper, we present model-based prioritization for a class of modifications for which models are not modified (only the source code is modified). After identifying the elements of the model related to the modified source code, information collected during the execution of the model is used to prioritize tests for execution. Here, we present and compare existing and new model-based test prioritization methods focused on this class of modifications. The major motivation for presenting these methods is to provide system developers with simple and yet effective test prioritization techniques for early fault detection. Statistical analysis of the empirical study, which compares the effectiveness of the presented methods in terms of early fault detection, show that compared to random ordering of test cases, model-based test prioritization significantly improve the effectiveness of test prioritization with respect to early fault detection.

[1]  Ahmed E. Hassan,et al.  Static test case prioritization using topic models , 2014, Empirical Software Engineering.

[2]  Chayanika Sharma,et al.  Prioritization of test case scenarios derived from activity diagram using genetic algorithm , 2010, 2010 International Conference on Computer and Communication Technology (ICCCT).

[3]  Mark Harman,et al.  Search Algorithms for Regression Test Case Prioritization , 2007, IEEE Transactions on Software Engineering.

[4]  A. Orso,et al.  Retesting software during development and maintenance , 2008, 2008 Frontiers of Software Maintenance.

[5]  Gregg Rothermel,et al.  An empirical study of the effects of minimization on the fault detection capabilities of test suites , 1998, Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272).

[6]  A. Jefferson Offutt,et al.  Maintaining evolving component-based software with UML , 2003, Seventh European Conference onSoftware Maintenance and Reengineering, 2003. Proceedings..

[7]  Neil Walkinshaw,et al.  Automated discovery of state transitions and their functions in source code , 2008, Softw. Test. Verification Reliab..

[8]  Lionel C. Briand,et al.  Achieving scalable model-based testing through test case diversity , 2013, TSEM.

[9]  Gregg Rothermel,et al.  An experimental determination of sufficient mutant operators , 1996, TSEM.

[10]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[11]  Gregg Rothermel,et al.  Test case prioritization: an empirical study , 1999, Proceedings IEEE International Conference on Software Maintenance - 1999 (ICSM'99). 'Software Maintenance for Business Change' (Cat. No.99CB36360).

[12]  Bogdan Korel,et al.  Model based regression test reduction using dependence analysis , 2002, International Conference on Software Maintenance, 2002. Proceedings..

[13]  Mehrdad Saadatmand,et al.  Mapping of State Machines to Code: Potentials and Challenges , 2014, ICSEA 2014.

[14]  Yanping Chen,et al.  Specification-based regression test selection with risk analysis , 2002, CASCON.

[15]  Gregg Rothermel,et al.  Incorporating varying test costs and fault severities into test case prioritization , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[16]  Lionel C. Briand,et al.  Automating impact analysis and regression test selection based on UML designs , 2002, International Conference on Software Maintenance, 2002. Proceedings..

[17]  J. D. de Winter.  Using the Student ’ s t-test with extremely small sample sizes , 2013 .

[18]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[19]  Amitabh Srivastava,et al.  Effectively prioritizing tests in development environment , 2002, ISSTA '02.

[20]  Mark Harman,et al.  Regression Testing Minimisation, Selection and Prioritisation - A Survey , 2009 .

[21]  Rajiv Gupta,et al.  An approach to regression testing using slicing , 1992, Proceedings Conference on Software Maintenance 1992.

[22]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[23]  Orest Pilskalns,et al.  Regression Testing UML Designs , 2006, 2006 22nd IEEE International Conference on Software Maintenance.

[24]  Jeremy Dick,et al.  Automating the Generation and Sequencing of Test Cases from Model-Based Specifications , 1993, FME.

[25]  Ritu Sibal,et al.  Prioritization of test scenarios derived from UML activity diagram using path complexity , 2012, CUBE.

[26]  Adam A. Porter,et al.  A history-based test prioritization technique for regression testing in resource constrained environments , 2002, ICSE '02.

[27]  Gregg Rothermel,et al.  Test Case Prioritization: A Family of Empirical Studies , 2002, IEEE Trans. Software Eng..

[28]  Joe D. Warren,et al.  The program dependence graph and its use in optimization , 1984, TOPL.

[29]  Zafar I. Malik,et al.  An approach for selective state machine based regression testing , 2007, A-MOST '07.

[30]  Doo-Hwan Bae,et al.  A test sequence selection method for statecharts , 2000 .

[31]  Neil Walkinshaw,et al.  Automated discovery of state transitions and their functions in source code , 2008 .

[32]  Gregg Rothermel,et al.  Selecting tests and identifying test coverage requirements for modified software , 1994, ISSTA '94.

[33]  Taghi M. Khoshgoftaar,et al.  Application of neural networks to software quality modeling of a very large telecommunications system , 1997, IEEE Trans. Neural Networks.

[34]  Gregg Rothermel,et al.  A safe, efficient regression test selection technique , 1997, TSEM.

[35]  Monalisa Sarma,et al.  System testing for object-oriented systems with test case prioritization , 2009 .

[36]  Bogdan Korel,et al.  Dependence analysis in reduction of requirement based test suites , 2002, ISSTA '02.

[37]  Bogdan Korel,et al.  Application of system models in regression test suite prioritization , 2008, 2008 IEEE International Conference on Software Maintenance.

[38]  Bogdan Korel,et al.  Model-based test prioritization heuristic methods and their evaluation , 2007, A-MOST '07.

[39]  Gregg Rothermel,et al.  Using component metadata to regression test component‐based software , 2007, Softw. Test. Verification Reliab..

[40]  Bogdan Korel,et al.  Experimental Comparison of Code-Based and Model-Based Test Prioritization , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[41]  Amy S. Beavers,et al.  Practical Considerations for Using Exploratory Factor Analysis in Educational Research. , 2013 .

[42]  Jean-Marc Jézéquel,et al.  Efficient object-oriented integration and regression testing , 2000, IEEE Trans. Reliab..

[43]  T. H. Tse,et al.  Adaptive Random Test Case Prioritization , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[44]  Eda Marchetti,et al.  The Cow_Suite Approach to Planning and Deriving Test Suites in UML Projects , 2002, UML.

[45]  Lee J. White,et al.  Test Manager: A regression testing tool , 1993, 1993 Conference on Software Maintenance.

[46]  Mark Harman,et al.  Test prioritization using system models , 2005, 21st IEEE International Conference on Software Maintenance (ICSM'05).

[47]  Kwang-Ting Cheng,et al.  Automatic Functional Test Generation Using The Extended Finite State Machine Model , 1993, 30th ACM/IEEE Design Automation Conference.

[48]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[49]  Mark Harman,et al.  Clustering test cases to achieve effective and scalable prioritisation incorporating expert knowledge , 2009, ISSTA.

[50]  Mark Harman,et al.  Control Dependence for Extended Finite State Machines , 2009, FASE.

[51]  Joseph Robert Horgan,et al.  Test set size minimization and fault detection effectiveness: a case study in a space application , 1997, Proceedings Twenty-First Annual International Computer Software and Applications Conference (COMPSAC'97).

[52]  Gregg Rothermel,et al.  A Static Approach to Prioritizing JUnit Test Cases , 2012, IEEE Transactions on Software Engineering.

[53]  Taehyung Wang,et al.  Model-based testing and maintenance , 2004, IEEE Sixth International Symposium on Multimedia Software Engineering.

[54]  Gregg Rothermel,et al.  On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques , 2006, IEEE Transactions on Software Engineering.

[55]  Deepti Mishra,et al.  Test case prioritization: a systematic mapping study , 2012, Software Quality Journal.

[56]  Mark Harman,et al.  A theoretical and empirical study of EFSM dependence , 2009, 2009 IEEE International Conference on Software Maintenance.

[57]  T. H. Tse,et al.  A Subsumption Hierarchy of Test Case Prioritization for Composite Services , 2015, IEEE Transactions on Services Computing.

[58]  Lionel C. Briand,et al.  Automating regression test selection based on UML designs , 2009, Inf. Softw. Technol..

[59]  John G. van Bosse,et al.  Digital Subscriber Signaling System No. 1 , 2002 .

[60]  Xiang Li,et al.  Assuring the model evolution of protocol software specifications by regression testing process improvement , 2011, Softw. Pract. Exp..

[61]  Mark Harman,et al.  Regression test suite prioritization using system models , 2012, Softw. Test. Verification Reliab..

[62]  T. H. Tse,et al.  XML-manipulating test case prioritization for XML-manipulating services , 2011, J. Syst. Softw..

[63]  Dieter Hogrefe,et al.  The CCITT-Specification and Description Language SDL , 1989, Comput. Networks.

[64]  Yanping Chen,et al.  Model-based regression test suite generation using dependence analysis , 2007, A-MOST '07.

[65]  Gordon Fraser,et al.  Test-case prioritization with model-checkers , 2007 .

[66]  Joseph Robert Horgan,et al.  A study of effective regression testing in practice , 1997, Proceedings The Eighth International Symposium on Software Reliability Engineering.

[67]  Yanping Chen,et al.  Regression test suite reduction based on SDL models of system requirements , 2009, J. Softw. Maintenance Res. Pract..

[68]  Bo Jiang,et al.  Prioritizing Test Cases for Regression Testing of Location-Based Services: Metrics, Techniques, and Case Study , 2014, IEEE Transactions on Services Computing.

[69]  Nikolai Tillmann,et al.  A Model-to-Implementation Mapping Tool for Automated Model-Based GUI Testing , 2005, ICFEM.

[70]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[71]  Henry Muccini,et al.  Reasoning about software architecture-based regression testing through a case study , 2005, 29th Annual International Computer Software and Applications Conference (COMPSAC'05).

[72]  Hasan Ural,et al.  Regression test suite selection using dependence analysis , 2013, J. Softw. Evol. Process..

[73]  Hüsnü Yenigün.  Identifying the effects of modifications as data dependencies , 2013, Software Quality Journal.