Scope-aided test prioritization, selection and minimization for software reuse

Abstract Software reuse can improve productivity, but does not exempt developers from the need to test the reused code into the new context. For this purpose, we propose here specific approaches to white-box test prioritization, selection and minimization that take into account the reuse context when reordering or selecting test cases, by leveraging possible constraints delimiting the new input domain scope. Our scope-aided testing approach aims at detecting those faults that under such constraints would be more likely triggered in the new reuse context, and is proposed as a boost to existing approaches. Our empirical evaluation shows that in test suite prioritization we can improve the average rate of faults detected when considering faults that are in scope, while remaining competitive considering all faults; in test case selection and minimization we can considerably reduce the test suite size, with small to no extra impact on fault detection effectiveness considering both in-scope and all faults. Indeed, in minimization, we improve the in-scope fault detection effectiveness in all cases.

[1]  Gregg Rothermel,et al.  A Unified Test Case Prioritization Approach , 2014, ACM Trans. Softw. Eng. Methodol..

[2]  Reid Holmes,et al.  Coverage is not strongly correlated with test suite effectiveness , 2014, ICSE.

[3]  Rajiv Gupta,et al.  A methodology for controlling the size of a test suite , 1990, Proceedings. Conference on Software Maintenance 1990.

[4]  Daniel Sundmark,et al.  Reuse with Software Components - A Survey of Industrial State of Practice , 2009, ICSR.

[5]  Reidar Conradi,et al.  An empirical study of software reuse vs. defect-density and stability , 2004, Proceedings. 26th International Conference on Software Engineering.

[6]  Marcus A. Rothenberger,et al.  Software reuse strategies and component markets , 2003, CACM.

[7]  Elaine J. Weyuker,et al.  An Applicable Family of Data Flow Testing Criteria , 1988, IEEE Trans. Software Eng..

[8]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[9]  David Notkin,et al.  An empirical study of static call graph extractors , 1998, TSEM.

[10]  Deepti Mishra,et al.  Test case prioritization: a systematic mapping study , 2012, Software Quality Journal.

[11]  T. Y. Chen,et al.  Heuristics Towards The Optimization Of TheSize Of A Test Suite , 1970 .

[12]  Gregg Rothermel,et al.  Test case prioritization: an empirical study , 1999, Proceedings IEEE International Conference on Software Maintenance - 1999 (ICSM'99). 'Software Maintenance for Business Change' (Cat. No.99CB36360).

[13]  Mary Lou Soffa,et al.  A methodology for controlling the size of a test suite , 1993, TSEM.

[14]  Juha Taina,et al.  Product family testing: a survey , 2004, SOEN.

[15]  Gregg Rothermel,et al.  Incorporating varying test costs and fault severities into test case prioritization , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[16]  Gordon Fraser,et al.  Redundancy Based Test-Suite Reduction , 2007, FASE.

[17]  Yue Jia,et al.  MILU: A Customizable, Runtime-Optimized Higher Order Mutation Testing Tool for the Full C Language , 2008, Testing: Academic & Industrial Conference - Practice and Research Techniques (taic part 2008).

[18]  Mary Jean Harrold,et al.  Test-suite reduction and prioritization for modified condition/decision coverage , 2001, Proceedings IEEE International Conference on Software Maintenance. ICSM 2001.

[19]  David S. Johnson,et al.  Approximation algorithms for combinatorial problems , 1973, STOC.

[20]  Corina S. Pasareanu,et al.  A survey of new trends in symbolic execution for software testing and analysis , 2009, International Journal on Software Tools for Technology Transfer.

[21]  Elaine J. Weyuker,et al.  Testing Component-Based Software: A Cautionary Tale , 1998, IEEE Softw..

[22]  Mark Harman,et al.  Search Algorithms for Regression Test Case Prioritization , 2007, IEEE Transactions on Software Engineering.

[23]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[24]  Nancy G. Leveson,et al.  An investigation of the Therac-25 accidents , 1993, Computer.

[25]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[26]  David S. Johnson Approximation algorithms for combinatorial problems , 1973, STOC '73.

[27]  Ossi Taipale,et al.  Test case selection and prioritization: risk-based or design-based? , 2010, ESEM '10.

[28]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[29]  Per Runeson,et al.  Software product line testing - A systematic mapping study , 2011, Inf. Softw. Technol..

[30]  Thomas J. Ostrand,et al.  Automatic generation of test scripts from formal test specifications , 1989 .

[31]  Adam A. Porter,et al.  Testing component compatibility in evolving configurations , 2013, Inf. Softw. Technol..

[32]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[33]  Richard Torkar,et al.  A survey on testing and reuse , 2003, Proceedings 2003 Symposium on Security and Privacy.

[34]  Vahid Garousi,et al.  A replicated survey of software testing practices in the Canadian province of Alberta: What has changed from 2004 to 2009? , 2010, J. Syst. Softw..

[35]  Ting Chen,et al.  State of the art: Dynamic symbolic execution for automated test generation , 2013, Future Gener. Comput. Syst..

[36]  Rubén Prieto-Díaz,et al.  Status report: software reusability , 1993, IEEE Software.

[37]  Jeffrey S. Poulin,et al.  The Business Case for Software Reuse , 1993, IBM Syst. J..

[38]  Mark Harman Making the Case for MORTO: Multi Objective Regression Test Optimization , 2011, 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.

[39]  Andrea Polini,et al.  Testing software components for integration: a survey of issues and techniques , 2007, Softw. Test. Verification Reliab..

[40]  Jerry Zeyu Gao,et al.  Testing Component-Based Software - Issues, Challenges, and Solutions , 2004, ICCBSS.

[41]  Koushik Sen,et al.  Symbolic execution for software testing: three decades later , 2013, CACM.

[42]  Sadahiro Isoda,et al.  Success factors of systematic reuse , 1994, IEEE Software.

[43]  Antonia Bertolino,et al.  Improving Test Coverage Measurement for Reused Software , 2015, 2015 41st Euromicro Conference on Software Engineering and Advanced Applications.

[44]  John D. Musa,et al.  Software-Reliability-Engineered Testing , 1996, Computer.

[45]  T. H. Tse,et al.  Adaptive Random Test Case Prioritization , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[46]  Maurizio Morisio,et al.  Success and Failure Factors in Software Reuse , 2002, IEEE Trans. Software Eng..

[47]  Gregg Rothermel,et al.  An empirical study of regression testing techniques incorporating context and lifetime factors and improved cost-benefit models , 2006, SIGSOFT '06/FSE-14.

[48]  Alessandro Orso,et al.  Component Metadata for Software Engineering Tasks , 2000, EDO.

[49]  Thomas J. Ostrand,et al.  Automatic Generation of Test Scripts from Formal Test Specifications , 1989, Symposium on Testing, Analysis, and Verification.