A Case Study on Robustness Fault Characteristics for Combinatorial Testing - Results and Challenges

Combinatorial testing is a well-known black-box testing approach. Empirical studies suggest the effectiveness of combinatorial coverage criteria. So far, the research focuses on positive test scenarios. But, robustness is an important characteristic of software systems and testing negative scenarios is crucial. Combinatorial strategies are extended to generate invalid test inputs but the effectiveness of negative test scenarios is yet unclear. Therefore, we conduct a case study and analyze 434 failures reported as bugs of an financial enterprise application. As a result, 51 robustness failures are identified including failures triggered by invalid value combinations and failures triggered by interactions of valid and invalid values. Based on the findings, four challenges for combinatorial robustness testing are derived. Keywords-Software Testing, Combinatorial Testing, Robustness Testing, Test Design

[1]  Per Runeson,et al.  Guidelines for conducting and reporting case study research in software engineering , 2009, Empirical Software Engineering.

[2]  Horst Lichter,et al.  Combinatorial Testing with Constraints for Negative Test Cases , 2018, 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[3]  Standard Glossary of Software Engineering Terminology , 1990 .

[4]  Myra B. Cohen,et al.  Reducing Masking Effects in CombinatorialInteraction Testing: A Feedback DrivenAdaptive Approach , 2014, IEEE Transactions on Software Engineering.

[5]  Michael L. Fredman,et al.  The AETG System: An Approach to Testing Based on Combinatiorial Design , 1997, IEEE Trans. Software Eng..

[6]  Jacek Czerwonka,et al.  Pairwise Testing in Real World , 2006 .

[7]  A. Jefferson Offutt,et al.  Combination testing strategies: a survey , 2005, Softw. Test. Verification Reliab..

[8]  Daniel Sundmark,et al.  Effects of Negative Testing on TDD: An Industrial Experiment , 2013, XP.

[9]  Raghu Kacker,et al.  The Relationship between Software Bug Type and Number of Factors Involved in Failures , 2016, 2016 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW).

[10]  Wasif Afzal,et al.  Testability and Software Robustness: A Systematic Literature Review , 2015, 2015 41st Euromicro Conference on Software Engineering and Advanced Applications.

[11]  Gerard Meszaros,et al.  xUnit Test Patterns: Refactoring Test Code , 2007 .

[12]  A. Jefferson Offutt,et al.  Test Oracle Strategies for Model-Based Testing , 2017, IEEE Transactions on Software Engineering.

[13]  Laura Marie Leventhal,et al.  Why software testing is sometimes ineffective: two applied studies of positive test strategy , 1994 .

[14]  Rachel Tzoref,et al.  System Level Combinatorial Testing in Practice -- The Concurrent Maintenance Case Study , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[15]  Jeff Yu Lei,et al.  ACTS: A Combinatorial Test Generation Tool , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[16]  Kishor S. Trivedi,et al.  How do bugs surface? A comprehensive study on the characteristics of software bugs manifestation , 2016, J. Syst. Softw..

[17]  D. Richard Kuhn,et al.  FAILURE MODES IN MEDICAL DEVICE SOFTWARE: AN ANALYSIS OF 15 YEARS OF RECALL DATA , 2001 .

[18]  Laura M. Leventhal,et al.  Positive Test Bias in Software Testing Among Professionals: A Review , 1993, EWHCI.

[19]  D. Richard Kuhn,et al.  Pseudo-Exhaustive Testing for Software , 2006, 2006 30th Annual IEEE/NASA Software Engineering Workshop.

[20]  D. Richard Kuhn,et al.  Software fault interactions and implications for software testing , 2004, IEEE Transactions on Software Engineering.

[21]  Jeff Yu Lei,et al.  Estimating t-Way Fault Profile Evolution During Testing , 2016, 2016 IEEE 40th Annual Computer Software and Applications Conference (COMPSAC).

[22]  Rachel Tzoref-Brill,et al.  Advances in Combinatorial Testing , 2019, Advances in Computers.

[23]  Daniel P. Siewiorek,et al.  A dimensionality model approach to testing and improving software robustness , 1999, 1999 IEEE AUTOTESTCON Proceedings (Cat. No.99CH36323).

[24]  Mark Harman,et al.  The Oracle Problem in Software Testing: A Survey , 2015, IEEE Transactions on Software Engineering.

[25]  A. Jefferson Offutt,et al.  An industrial study of applying input space partitioning to test financial calculation engines , 2014, Empirical Software Engineering.

[26]  M. J. Reilly,et al.  An investigation of the applicability of design of experiments to software testing , 2002, 27th Annual NASA Goddard/IEEE Software Engineering Workshop, 2002. Proceedings..

[27]  M.A. Vouk,et al.  On effectiveness of pairwise methodology for testing network-centric software , 2005, 2005 International Conference on Information and Communication Technology.