Combinatorial Coverage Difference Measurement

Structural coverage criteria are widely used tools in software engineering, useful for measuring aspects of test execution thoroughness. However, in many cases structural coverage may not be applicable, either because source code is not available, or because processing is based on neural networks or other black-box components. Vulnerability and fault detection in such cases will typically rely on large volumes of tests, with the goal of discovering flaws that result in system failures or security weaknesses. This publication explains combinatorial coverage difference measures that have been applied to problems that include fault identification and autonomous systems validation, and documents functions of research tools for computing these measures. The metrics and tools described are introduced as research tools; later work will be useful in determining which are of value in assurance and testing or simulation.

[1]  Raghu Kacker,et al.  TLS Cipher Suites Recommendations: A Combinatorial Coverage Measurement Approach , 2016, 2016 IEEE International Conference on Software Quality, Reliability and Security (QRS).

[2]  Paolo Arcaini,et al.  Efficient and Guaranteed Detection of t-Way Failure-Inducing Combinations , 2019, 2019 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[3]  Charles J. Colbourn,et al.  On a Combinatorial Framework for Fault Characterization , 2018, Math. Comput. Sci..

[4]  Raghu Kacker,et al.  Combinatorial Methods in Security Testing , 2016, Computer.

[5]  Rachel Tzoref,et al.  Practical Fault Localization with Combinatorial Test Design , 2019, 2019 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[6]  Raghu Kacker,et al.  The Relationship between Software Bug Type and Number of Factors Involved in Failures , 2016, 2016 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW).

[7]  Yu Lei,et al.  A Combinatorial Testing-Based Approach to Fault Localization , 2020, IEEE Transactions on Software Engineering.

[8]  D. Richard Kuhn,et al.  Software fault interactions and implications for software testing , 2004, IEEE Transactions on Software Engineering.

[9]  Eduard Paul Enoiu,et al.  On Measuring Combinatorial Coverage of Manually Created Test Cases for Industrial Software , 2019, 2019 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[10]  R. Krishnan,et al.  Approaches to Fault Localization in Combinatorial Testing: A Survey , 2018 .

[11]  Jeff Yu Lei,et al.  Combinatorial Coverage Measurement Concepts and Applications , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops.

[12]  Dong Li,et al.  Applying Combinatorial Testing in Industrial Settings , 2016, 2016 IEEE International Conference on Software Quality, Reliability and Security (QRS).

[13]  Dimitris E. Simos,et al.  CAMETRICS: A Tool for Advanced Combinatorial Analysis and Measurement of Test Sets , 2018, 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[14]  Kishor S. Trivedi,et al.  How do bugs surface? A comprehensive study on the characteristics of software bugs manifestation , 2016, J. Syst. Softw..

[15]  Jeff Yu Lei,et al.  BEN: A combinatorial testing-based fault localization tool , 2015, 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[16]  D. Richard Kuhn,et al.  Pseudo-Exhaustive Testing for Software , 2006, 2006 30th Annual IEEE/NASA Software Engineering Workshop.

[17]  Yu Lei,et al.  Practical Combinatorial Testing , 2010 .

[18]  Raghu N. Kacker,et al.  An Application of Combinatorial Methods for Explainability in Artificial Intelligence and Machine Learning (Draft) , 2019 .

[19]  D. Richard Kuhn,et al.  Combinatorial Testing Metrics for Machine Learning , 2021, 2021 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[20]  Raghu Kacker,et al.  A method for analyzing system state-space coverage within a t-wise testing framework , 2010, 2010 IEEE International Systems Conference.

[21]  Yu Lei,et al.  Combinatorial coverage as an aspect of test quality , 2015 .

[22]  M. J. Reilly,et al.  An investigation of the applicability of design of experiments to software testing , 2002, 27th Annual NASA Goddard/IEEE Software Engineering Workshop, 2002. Proceedings..

[23]  Elaine J. Weyuker,et al.  On Testing Non-Testable Programs , 1982, Comput. J..

[24]  Murat Ozcan,et al.  Applications of Practical Combinatorial Testing Methods at Siemens Industry Inc., Building Technologies Division , 2017, 2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[25]  Hong Zhu,et al.  Software unit test coverage and adequacy , 1997, ACM Comput. Surv..

[26]  Neena Goveas,et al.  Test Suite Design Methodology Using Combinatorial Approach for Internet of Things Operating Systems , 2015 .