Combinatorial Testing in an Industrial Environment -- Analyzing the Applicability of a Tool

Numerous combinatorial testing tools are available for generating test cases. However, many of them are never used in practice. One of the reasons is the lack of empirical studies that involve human subjects applying testing techniques. This paper aims to investigate the applicability of a combinatorial testing tool in the company SOFTEAM. A case study is designed and conducted within the development team responsible for a new product. The participants consist of 3 practitioners from the company. The applicability of the tool has been examined in terms of efficiency, effectiveness and learning effort.

[1]  Daniel Sundmark,et al.  A Framework for Comparing Efficiency, Effectiveness and Applicability of Software Testing Techniques , 2006, Testing: Academic & Industrial Conference - Practice And Research Techniques (TAIC PART'06).

[2]  Hareton K. N. Leung,et al.  A survey of combinatorial testing , 2011, CSUR.

[3]  Joachim Wegener,et al.  Test Sequence Generation from Classification Trees , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[4]  Tanja E. J. Vos,et al.  The OMG UML Testing Profile in Use--An Industrial Case Study for the Future Internet Testing , 2013, 2013 17th European Conference on Software Maintenance and Reengineering.

[5]  Tanja E. J. Vos,et al.  Combinatorial Testing Tool Learnability in an Industrial Environment , 2013, 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement.

[6]  Tanja E. J. Vos,et al.  Evaluating the FITTEST Automated Testing Tools: An Industrial Case Study , 2013, 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement.

[7]  Jeff Yu Lei,et al.  Combinatorial Software Testing , 2009, Computer.

[8]  L. C. Briand.  A Critical Analysis of Empirical Research in Software Testing , 2007, ESEM 2007.

[9]  Tanja E. J. Vos,et al.  Testing of future internet applications running in the cloud , 2013 .

[10]  Christopher M. Lott,et al.  Repeatable software engineering experiments for comparing defect-detection techniques , 2004, Empirical Software Engineering.

[11]  Nelly Condori-Fernandez,et al.  Evaluating applicability of combinatorial testing in an industrial environment: a case study , 2013, JAMAICA 2013.

[12]  Lionel C. Briand,et al.  Empirical studies of software testing techniques: challenges, practical strategies, and future research , 2004, SOEN.

[13]  Tanja E. J. Vos,et al.  FITTEST: A new continuous and automated testing process for future Internet applications , 2014, 2014 Software Evolution Week - IEEE Conference on Software Maintenance, Reengineering, and Reverse Engineering (CSMR-WCRE).

[14]  Matthias Grochtmann,et al.  Classification trees for partition testing , 1993, Softw. Test. Verification Reliab..

[15]  Tanja E. J. Vos,et al.  Evolutionary functional black-box testing in an industrial setting , 2013, Software Quality Journal.

[16]  Per Runeson,et al.  Guidelines for conducting and reporting case study research in software engineering , 2009, Empirical Software Engineering.

[17]  Tanja E. J. Vos,et al.  The FITTEST Tool Suite for Testing Future Internet Applications , 2013, FITTEST@ICTSS.

[18]  Beatriz Marín,et al.  A Methodological Framework for Evaluating Software Testing Techniques and Tools , 2012, 2012 12th International Conference on Quality Software.

[19]  K. Burr,et al.  Combinatorial Test Techniques : Table-based Automation , Test Generation and Code Coverage , 1998 .

[20]  Tanja E. J. Vos,et al.  Industrial Case Studies for Evaluating Search Based Structural Testing , 2012, Int. J. Softw. Eng. Knowl. Eng..

[21]  Giancarlo Succi,et al.  Effort Prediction in Iterative Software Development Processes -- Incremental Versus Global Prediction Models , 2007, ESEM 2007.

[22]  Tanja E. J. Vos.  Evolutionary Testing for Complex Systems , 2009, ERCIM News.

[23]  Peter M. Kruse,et al.  Automated Test Case Generation Using Classification Trees , 2010 .