Identifying Failure-Causing Schemas in the Presence of Multiple Faults

Combinatorial testing (CT) has been proven effective in revealing the failures caused by the interaction of factors that affect the behavior of a system. The theory of Minimal Failure-Causing Schema (MFS) has been proposed to isolate the cause of a failure after CT. Most algorithms that aim to identify MFS focus on handling a single fault in the System Under Test (SUT). However, we argue that multiple faults are more common in practice, under which masking effects may be triggered so that some failures cannot be observed. The traditional MFS theory lacks a mechanism to handle such effects; hence, they may incorrectly isolate the MFS. To address this problem, we propose a new MFS model that takes into account multiple faults. We first formally analyze the impact of the multiple faults on existing MFS identifying algorithms, especially in situations where masking effects are triggered by multiple faults. We then develop an approach that can assist traditional algorithms to better handle multiple faults. Empirical studies were conducted using several kinds of open-source software, which showed that multiple faults with masking effects do negatively affect traditional MFS identifying approaches and that our approach can help to alleviate these effects.

[1]  Hareton K. N. Leung,et al.  A survey of combinatorial testing , 2011, CSUR.

[2]  Jiangfan Shi,et al.  Exploiting Constraint Solving History to Construct Interaction Test Suites , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[3]  Myra B. Cohen,et al.  Interaction testing of highly-configurable systems in the presence of constraints , 2007, ISSTA '07.

[4]  Alex Groce,et al.  The Theory of Composite Faults , 2017, 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST).

[5]  Baowen Xu,et al.  A Software Debugging Method Based on Pairwise Testing , 2005, International Conference on Computational Science.

[6]  Angelo Gargantini,et al.  A Logic-Based Approach to Combinatorial Testing with Constraints , 2008, TAP.

[7]  Bin Wang,et al.  Automated support for classifying software failure reports , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[8]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[9]  David Leon,et al.  Finding failures by cluster analysis of execution profiles , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[10]  Myra B. Cohen,et al.  Reducing Masking Effects in CombinatorialInteraction Testing: A Feedback DrivenAdaptive Approach , 2014, IEEE Transactions on Software Engineering.

[11]  David W. Binkley,et al.  Program slicing , 2008, 2008 Frontiers of Software Maintenance.

[12]  Mary Jean Harrold,et al.  Debugging in Parallel , 2007, ISSTA '07.

[13]  Myra B. Cohen,et al.  Efficiency and early fault detection with lower and higher strength combinatorial interaction testing , 2013, ESEC/FSE 2013.

[14]  Hareton K. N. Leung,et al.  The Minimal Failure-Causing Schema of Combinatorial Testing , 2011, TSEM.

[15]  Renée C. Bryce,et al.  A framework of greedy methods for constructing interaction test suites , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[16]  Patrick J. Schroeder,et al.  Pairwise Testing : A Best Practice That Isn ’ , 2005 .

[17]  Charles J. Colbourn,et al.  Locating and detecting arrays for interaction faults , 2008, J. Comb. Optim..

[18]  Charles J. Colbourn,et al.  Prioritized interaction testing for pair-wise coverage with seeding and constraints , 2006, Inf. Softw. Technol..

[19]  Myra B. Cohen,et al.  Covering arrays for efficient fault characterization in complex configuration spaces , 2004, IEEE Transactions on Software Engineering.

[20]  Cemal Yilmaz Test Case-Aware Combinatorial Interaction Testing , 2013, IEEE Transactions on Software Engineering.

[21]  Jeff Yu Lei,et al.  Identifying Failure-Inducing Combinations Using Tuple Relationship , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops.

[22]  Lucia Moura,et al.  Algorithms to Locate Errors Using Covering Arrays , 2008, LATIN.

[23]  David Leon,et al.  Pursuing failure: the distribution of program failures in a profile space , 2001, ESEC/FSE-9.

[24]  Adam A. Porter,et al.  iTree: Efficiently Discovering High-Coverage Configurations Using Interaction Trees , 2014, IEEE Transactions on Software Engineering.

[25]  Charles Yang,et al.  Estimation of software reliability by stratified sampling , 1999, TSEM.

[26]  Baowen Xu,et al.  Adaptive Interaction Fault Location Based on Combinatorial Testing , 2010, 2010 10th International Conference on Quality Software.

[27]  Hareton K. N. Leung,et al.  Adaptive Combinatorial Testing , 2013, 2013 13th International Conference on Quality Software.

[28]  Alex Groce,et al.  Taming compiler fuzzers , 2013, ACM-SIGPLAN Symposium on Programming Language Design and Implementation.

[29]  Feng Duan,et al.  Constraint handling in combinatorial test generation using forbidden tuples , 2015, 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[30]  Myra B. Cohen,et al.  Incremental covering array failure characterization in large configuration spaces , 2009, ISSTA.

[31]  Myra B. Cohen,et al.  Augmenting simulated annealing to build interaction test suites , 2003, 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003..

[32]  Jeff Yu Lei,et al.  IPOG/IPOG‐D: efficient test generation for multi‐way combinatorial testing , 2008, Softw. Test. Verification Reliab..

[33]  Myra B. Cohen,et al.  Evaluating improvements to a meta-heuristic search for constrained interaction testing , 2011, Empirical Software Engineering.

[34]  Andreas Zeller,et al.  Simplifying and Isolating Failure-Inducing Input , 2002, IEEE Trans. Software Eng..

[35]  Charles Yang,et al.  Partition testing, stratified sampling, and cluster analysis , 1993, SIGSOFT '93.

[36]  Jie Li,et al.  Improved Delta Debugging Based on Combinatorial Testing , 2012, 2012 12th International Conference on Quality Software.

[37]  Myra B. Cohen,et al.  Constructing Interaction Test Suites for Highly-Configurable Systems in the Presence of Constraints: A Greedy Approach , 2008, IEEE Transactions on Software Engineering.

[38]  Raghu Kacker,et al.  Identifying Failure-Inducing Combinations in a Combinatorial Test Set , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[39]  Jonas Mellin,et al.  Handling Constraints in the Input Space when Using Combination Strategies for Software Testing , 2006 .

[40]  Jeff Yu Lei,et al.  Fault localization based on failure-inducing combinations , 2013, 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE).

[41]  Lucia Moura,et al.  Locating Errors Using ELAs, Covering Arrays, and Adaptive Testing Algorithms , 2009, SIAM J. Discret. Math..

[42]  Michael L. Fredman,et al.  The AETG System: An Approach to Testing Based on Combinatiorial Design , 1997, IEEE Trans. Software Eng..

[43]  Myra B. Cohen,et al.  Constructing test suites for interaction testing , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[44]  Myra B. Cohen,et al.  Feedback driven adaptive combinatorial testing , 2011, ISSTA '11.

[45]  Michael I. Jordan,et al.  Statistical debugging: simultaneous identification of multiple bugs , 2006, ICML.

[46]  Jeff Yu Lei,et al.  Isolating Failure-Inducing Combinations in Combinatorial Testing Using Test Augmentation and Classification , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[47]  Lior Rokach,et al.  Top-down induction of decision trees classifiers - a survey , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[48]  Jian Zhang,et al.  Characterizing failure-causing parameter interactions by adaptive testing , 2011, ISSTA '11.

[49]  James A. Jones,et al.  Software Behavior and Failure Clustering: An Empirical Study of Fault Causality , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.