Advances in Combinatorial Testing

Abstract Since their introduction into software testing in the mid-1980s, combinatorial methods for test design gathered popularity as a testing best practice and as a prominent software testing research area. This chapter reviews recent advances in combinatorial testing, with special focus on the research since 2011. It provides a brief background on the theory behind combinatorial testing and on its use in practice. Requirements from industry usage have led to advances in various areas examined in this chapter, including constraints handling in combinatorial algorithms, support for the combinatorial modeling process, and studies on metrics to support the effectiveness of combinatorial testing. We also highlight recent case studies describing novel use cases for test and field quality improvement in the context of system test, and for optimization of test data. Finally, we examine recent developments in advanced topics such as utilization of existing tests, test case prioritization, fault localization, and evolution of combinatorial models.

[1]  M. J. Reilly,et al.  An investigation of the applicability of design of experiments to software testing , 2002, 27th Annual NASA Goddard/IEEE Software Engineering Workshop, 2002. Proceedings..

[2]  Angelo Gargantini,et al.  Using decision trees to aid algorithm selection in combinatorial interaction tests generation , 2015, 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[3]  Neelu Kumari,et al.  Combinatorial Testing for an Automotive Hybrid Electric Vehicle Control System: A Case Study , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops.

[4]  M.A. Vouk,et al.  On effectiveness of pairwise methodology for testing network-centric software , 2005, 2005 International Conference on Information and Communication Technology.

[5]  Armin Biere,et al.  Greedy combinatorial test case generation using unsatisfiable cores , 2016, 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE).

[6]  Alan Hartman,et al.  Problems and algorithms for covering arrays , 2004, Discret. Math..

[7]  Raghu Kacker,et al.  Introducing Combinatorial Testing in a Large Organization: Pilot Project Experience Report , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops.

[8]  D. Richard Kuhn,et al.  FAILURE MODES IN MEDICAL DEVICE SOFTWARE: AN ANALYSIS OF 15 YEARS OF RECALL DATA , 2001 .

[9]  Hareton K. N. Leung,et al.  A survey of combinatorial testing , 2011, CSUR.

[10]  R. Fisher.  Statistical methods for research workers , 1927, Protoplasma.

[11]  Eitan Farchi,et al.  Using Projections to Debug Large Combinatorial Models , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops.

[12]  Shahar Maoz,et al.  Visualization of combinatorial models and test plans , 2016, 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE).

[13]  David Anderson,et al.  Relationship between pair-wise and MC/DC testing: Initial experimental results , 2015, 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[14]  Michael L. Fredman,et al.  The AETG System: An Approach to Testing Based on Combinatiorial Design , 1997, IEEE Trans. Software Eng..

[15]  D.M. Cohen,et al.  The Combinatorial Design Approach to Automatic Test Generation , 1996, IEEE Softw..

[16]  Myra B. Cohen,et al.  Covering arrays for efficient fault characterization in complex configuration spaces , 2004, IEEE Transactions on Software Engineering.

[17]  R. A. Fisher,et al.  Design of Experiments , 1936 .

[18]  Myra B. Cohen,et al.  Practical Combinatorial Interaction Testing: Empirical Findings on Efficiency and Early Fault Detection , 2015, IEEE Transactions on Software Engineering.

[19]  Jeff Yu Lei,et al.  Identifying Failure-Inducing Combinations Using Tuple Relationship , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops.

[20]  Myra B. Cohen,et al.  Evaluating improvements to a meta-heuristic search for constrained interaction testing , 2011, Empirical Software Engineering.

[21]  Itai Segall,et al.  Common Patterns in Combinatorial Models , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[22]  José Torres-Jiménez,et al.  A Max-SAT-Based Approach to Constructing Optimal Covering Arrays , 2013, CCIA.

[23]  Jeff Yu Lei,et al.  Estimating t-Way Fault Profile Evolution During Testing , 2016, 2016 IEEE 40th Annual Computer Software and Applications Conference (COMPSAC).

[24]  Yu Lei,et al.  Applying combinatorial test data generation to big data applications , 2016, 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE).

[25]  Yu Lei,et al.  In-parameter-order: a test generation strategy for pairwise testing , 1998, Proceedings Third IEEE International High-Assurance Systems Engineering Symposium (Cat. No.98EX231).

[26]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[27]  Ajitha Rajan,et al.  Interaction Testing in Model-Based Development: Effect on Model-Coverage , 2006, 2006 13th Asia Pacific Software Engineering Conference (APSEC'06).

[28]  Krishnan Rangarajan,et al.  Deriving Combinatorial Test Design Model from UML Activity Diagram , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops.

[29]  Macario Polo,et al.  Test Case Generation with Regular Expressions and Combinatorial Techniques , 2017, 2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[30]  Myra B. Cohen,et al.  Combinatorial Interaction Regression Testing: A Study of Test Case Generation and Prioritization , 2007, 2007 IEEE International Conference on Software Maintenance.

[31]  Patrick J. Schroeder,et al.  Comparing the fault detection effectiveness of n-way and random test suites , 2004, Proceedings. 2004 International Symposium on Empirical Software Engineering, 2004. ISESE '04..

[32]  Myra B. Cohen,et al.  A Study in Prioritization for Higher Strength Combinatorial Testing , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops.

[33]  Jeff Yu Lei,et al.  An Empirical Comparison of Combinatorial and Random Testing , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops.

[34]  Tatsuhiro Tsuchiya,et al.  Applicability of non-specification-based approaches to logic testing for software , 2001, 2001 International Conference on Dependable Systems and Networks.

[35]  Feng Duan,et al.  Constraint handling in combinatorial test generation using forbidden tuples , 2015, 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[36]  Myra B. Cohen,et al.  Learning Combinatorial Interaction Test Generation Strategies Using Hyperheuristic Search , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[37]  Jeff Yu Lei,et al.  Practical Combinatorial Testing: Beyond Pairwise , 2008, IT Professional.

[38]  Jeff Yu Lei,et al.  Evaluating the Effectiveness of BEN in Localizing Different Types of Software Fault , 2016, 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[39]  Jeff Yu Lei,et al.  An Efficient Algorithm for Constraint Handling in Combinatorial Test Generation , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[40]  Jeff Yu Lei,et al.  BEN: A combinatorial testing-based fault localization tool , 2015, 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[41]  Itai Segall,et al.  Combining minimization and generation for combinatorial testing , 2015, 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[42]  Franz Wotawa,et al.  Mutation Score, Coverage, Model Inference: Quality Assessment for T-Way Combinatorial Test-Suites , 2017, 2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[43]  Tatsuhiro Tsuchiya,et al.  Using Satisfiability Solving for Pairwise Testing in the Presence of Constraints , 2012, IEICE Trans. Fundam. Electron. Commun. Comput. Sci..

[44]  Myra B. Cohen,et al.  Constructing test suites for interaction testing , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[45]  Yu Lei,et al.  Applying Combinatorial Testing to High-Speed Railway Track Circuit Receiver , 2017, 2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[46]  Ofer Shtrichman.  Pruning Techniques for the SAT-Based Bounded Model Checking Problem , 2001 .

[47]  Renée C. Bryce,et al.  A framework of greedy methods for constructing interaction test suites , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[48]  Shin-ichi Minato,et al.  Graph-Based Representations of Discrete Functions , 1996 .

[49]  Gadiel Seroussi,et al.  Vector sets for exhaustive testing of logic circuits , 1988, IEEE Trans. Inf. Theory.

[50]  Siddhartha R. Dalal,et al.  Model-based testing in practice , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[51]  John Joseph Chilenski,et al.  An Investigation of Three Forms of the Modified Condition Decision Coverage (MCDC) Criterion , 2001 .

[52]  Katsumi Inoue,et al.  Generating Combinatorial Test Cases by Efficient SAT Encodings Suitable for CDCL SAT Solvers , 2010, LPAR.

[53]  Krishnan Rangarajan,et al.  Extracting the Combinatorial Test Parameters and Values from UML Sequence Diagrams , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops.

[54]  Sergiy Vilkomir,et al.  Using Combinatorial Approaches for Testing Mobile Applications , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops.

[55]  George B. Sherwood.  Embedded Functions for Constraints and Variable Strength in Combinatorial Testing , 2016, 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[56]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[57]  Myra B. Cohen,et al.  Constructing Interaction Test Suites for Highly-Configurable Systems in the Presence of Constraints: A Greedy Approach , 2008, IEEE Transactions on Software Engineering.

[58]  Raghu Kacker,et al.  Identifying Failure-Inducing Combinations in a Combinatorial Test Set , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[59]  R. Krishnan,et al.  Combinatorial testing: learnings from our experience , 2007, SOEN.

[60]  Sachin Jain,et al.  A tool for combination-based prioritization and reduction of user-session-based test suites , 2011, 2011 27th IEEE International Conference on Software Maintenance (ICSM).

[61]  Myra B. Cohen,et al.  A variable strength interaction testing of components , 2003, Proceedings 27th Annual International Computer Software and Applications Conference. COMPAC 2003.

[62]  Brent Hailpern,et al.  Software debugging, testing, and verification , 2002, IBM Syst. J..

[63]  Colin L. Mallows,et al.  Factor-covering designs for testing software , 1998 .

[64]  Hironori Washizaki,et al.  Pairwise Coverage-Based Testing with Selected Elements in a Query for Database Applications , 2016, 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[65]  Jacek Czerwonka,et al.  On Use of Coverage Metrics in Assessing Effectiveness of Combinatorial Test Designs , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops.

[66]  Angelo Gargantini,et al.  A Formal Logic Approach to Constrained Combinatorial Testing , 2010, Journal of Automated Reasoning.

[67]  Shahar Maoz,et al.  Lattice-Based Semantics for Combinatorial Model Evolution , 2015, ATVA.

[68]  George B. Sherwood.  Embedded functions in combinatorial test designs , 2015, 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[69]  Yu Lei,et al.  Introduction to Combinatorial Testing , 2013 .

[70]  Sergiy Vilkomir,et al.  Evaluation of t-wise Approach for Testing Logical Expressions in Software , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops.

[71]  Myra B. Cohen,et al.  Reducing Masking Effects in CombinatorialInteraction Testing: A Feedback DrivenAdaptive Approach , 2014, IEEE Transactions on Software Engineering.

[72]  Orna Raz,et al.  Cluster-based test suite functional analysis , 2016, SIGSOFT FSE.

[73]  Itai Segall,et al.  Interactive refinement of combinatorial test plans , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[74]  Robert Mandl,et al.  Orthogonal Latin squares: an application of experiment design to compiler testing , 1985, CACM.

[75]  Tatsuhiro Tsuchiya,et al.  Towards Automatic Constraints Elicitation in Pair-Wise Testing Based on a Linguistic Approach: Elicitation Support Using Coupling Strength , 2015, 2015 IEEE/ACM 2nd International Workshop on Requirements Engineering and Testing.

[76]  Tatsuhiro Tsuchiya,et al.  Pairwise Testing in the Presence of Configuration Change Cost , 2008, 2008 Second International Conference on Secure System Integration and Reliability Improvement.

[77]  Myra B. Cohen,et al.  Interaction testing of highly-configurable systems in the presence of constraints , 2007, ISSTA '07.

[78]  Rachel Tzoref,et al.  System Level Combinatorial Testing in Practice -- The Concurrent Maintenance Case Study , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[79]  Armin Biere,et al.  Optimization of Combinatorial Testing by Incremental SAT Solving , 2015, 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST).

[80]  Jian Zhang,et al.  Characterizing failure-causing parameter interactions by adaptive testing , 2011, ISSTA '11.

[81]  Robert C. Martin.  Agile Software Development, Principles, Patterns, and Practices , 2002 .

[82]  Genichi Taguchi,et al.  Introduction to quality engineering.... , 2014 .

[83]  Fei-Ching Kuo,et al.  Test Suite Prioritization by Switching Cost , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops.

[84]  Paolo Arcaini,et al.  Validation of Models and Tests for Constrained Combinatorial Interaction Testing , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops.

[85]  Avi Ziv,et al.  Hole analysis for functional coverage data , 2002, DAC '02.

[86]  Itai Segall,et al.  Simplified Modeling of Combinatorial Test Spaces , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[87]  C. R. Rao,et al.  Factorial Experiments Derivable from Combinatorial Arrangements of Arrays , 1947 .

[88]  William Jenkins,et al.  Effectiveness of Pair-Wise Testing for Software with Boolean Inputs , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[89]  Itai Segall.  Repeated Combinatorial Test Design -- Unleashing the Potential in Multiple Testing Iterations , 2016, 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST).

[90]  Angelo Gargantini,et al.  CITLAB: A Laboratory for Combinatorial Interaction Testing , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[91]  Angelo Gargantini,et al.  Efficient Combinatorial Test Generation Based on Multivalued Decision Diagrams , 2014, Haifa Verification Conference.

[92]  N. Sloane.  Covering arrays and intersecting codes , 1993 .

[93]  Myra B. Cohen,et al.  Incremental covering array failure characterization in large configuration spaces , 2009, ISSTA.

[94]  Randal E. Bryant,et al.  Graph-Based Algorithms for Boolean Function Manipulation , 1986, IEEE Transactions on Computers.

[95]  Capers Jones.  Applied Software Measurement: Global Analysis of Productivity and Quality , 1991 .

[96]  Bernhard Garn,et al.  Combinatorial Methods for Modelling Composed Software Systems , 2017, 2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[97]  Shahar Maoz,et al.  Syntactic and Semantic Differencing for Combinatorial Models of Test Designs , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE).

[98]  Angelo Gargantini,et al.  Combinatorial Interaction Testing for Automated Constraint Repair , 2017, 2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[99]  Raghu Kacker,et al.  Combinatorial Methods in Security Testing , 2016, Computer.

[100]  Itai Segall,et al.  Interaction-based test-suite minimization , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[101]  Jez Humble,et al.  Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation , 2010 .

[102]  Ashish Jain,et al.  Modeling requirements for combinatorial software testing , 2005, ACM SIGSOFT Softw. Eng. Notes.

[103]  Darrel Ince,et al.  The Effectiveness of T-Way Test Data Generation , 2008, SAFECOMP.

[104]  Baowen Xu,et al.  A Software Debugging Method Based on Pairwise Testing , 2005, International Conference on Computational Science.

[105]  Steven David Prestwich,et al.  Constraint Models for the Covering Test Problem , 2006, Constraints.

[106]  Xi Deng,et al.  Combinatorial Testing on Implementations of HTML5 Support , 2017, 2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[107]  Frederick P. Brooks,et al.  The Mythical Man-Month: Essays on Softw , 1978 .

[108]  Yu Lei,et al.  Random vs. Combinatorial Methods for Discrete Event Simulation of a Grid Computer Network , 2009 .

[109]  Jeff Yu Lei,et al.  Combinatorial Coverage Measurement Concepts and Applications , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops.

[110]  Mark Harman,et al.  The Oracle Problem in Software Testing: A Survey , 2015, IEEE Transactions on Software Engineering.

[111]  Peter M. Kruse.  Test Oracles and Test Script Generation in Combinatorial Testing , 2016, 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[112]  Eitan Farchi,et al.  Using binary decision diagrams for combinatorial test design , 2011, ISSTA '11.

[113]  D. Richard Kuhn,et al.  Software fault interactions and implications for software testing , 2004, IEEE Transactions on Software Engineering.

[114]  Luisa Gargano,et al.  Capacities: From Information Theory to Extremal Set Theory , 1994, J. Comb. Theory, Ser. A.

[115]  Atif M. Memon,et al.  Developing a Single Model and Test Prioritization Strategies for Event-Driven Software , 2011, IEEE Transactions on Software Engineering.