Practical Combinatorial Testing

The Information Technology Laboratory (ITL) at the National Institute of Standards and Technology (NIST) promotes the U.S. economy and public welfare by providing technical leadership for the Nation's measurement and standards infrastructure. ITL develops tests, test methods, reference data, proof of concept implementations, and technical analyses to advance the development and productive use of information technology. ITL's responsibilities include the development of technical, physical, administrative, and management standards and guidelines for the cost-effective security and privacy of sensitive unclassified information in Federal computer systems. This Special Publication 800-series reports on ITL's research, guidance, and outreach efforts in computer security, and its collaborative activities with industry, government, and academic organizations. materials may be identified in this document in order to describe an experimental procedure or concept adequately. Such identification is not intended to imply recommendation or endorsement by the National Institute of Standards and Technology, nor is it intended to imply that the entities, materials, or equipment are necessarily the best available for the purpose. for early support and evangelism of this work, and especially Jim Lawrence who has been an integral part of the team since the beginning. We have benefitted tremendously from interactions with researchers and practitioners including who contributed to the software and methods described in this document.

[1]  Pedram Amini,et al.  Fuzzing: Brute Force Vulnerability Discovery , 2007 .

[2]  Fausto Giunchiglia,et al.  NUSMV: A New Symbolic Model Verifier , 1999, CAV.

[3]  Raghu Kacker,et al.  A method for analyzing system state-space coverage within a t-wise testing framework , 2010, 2010 IEEE International Systems Conference.

[4]  C. L. Mallows,et al.  Applying Design of Experiments to Software Testing , 1997, Proceedings of the (19th) International Conference on Software Engineering.

[5]  Standard Glossary of Software Engineering Terminology , 1990 .

[6]  John Joseph Chilenski,et al.  An Investigation of Three Forms of the Modified Condition Decision Coverage (MCDC) Criterion , 2001 .

[7]  Myra B. Cohen,et al.  Covering array sampling of input event sequences for automated gui testing , 2007, ASE.

[8]  Sigrid Eldh Software Testing Techniques , 2007 .

[9]  Jeffrey M. Voas,et al.  Putting assertions in their place , 1994, Proceedings of 1994 IEEE International Symposium on Software Reliability Engineering.

[10]  K. Burr,et al.  Combinatorial Test Techniques : Table-based Automation , Test Generation and Code Coverage , 1998 .

[11]  Tatsuhiro Tsuchiya,et al.  On fault classes and error detection capability of specification-based testing , 2002, TSEM.

[12]  Lee Copeland,et al.  A Practitioner's Guide to Software Test Design , 2003 .

[13]  Myra B. Cohen,et al.  Testing across configurations: implications for combinatorial testing , 2006, SOEN.

[14]  Paul Ammann,et al.  Abstracting formal specifications to generate software tests via model checking , 1999, Gateway to the New Millennium. 18th Digital Avionics Systems Conference. Proceedings (Cat. No.99CH37033).

[15]  Wenhua Wang,et al.  An Interaction-Based Test Sequence Generation Approach for Testing Web Applications , 2008, 2008 11th IEEE High Assurance Systems Engineering Symposium.

[16]  Bellanov S. Apilli Fault-based combinatorial testing of web services , 2009, OOPSLA Companion.

[17]  Colin L. Mallows,et al.  Factor-covering designs for testing software , 1998 .

[18]  Tao Xie,et al.  Property Verification for Generic Access Control Models , 2008, 2008 IEEE/IFIP International Conference on Embedded and Ubiquitous Computing.

[19]  David C. Luckham,et al.  An Overview of Anna, a Specification Language for Ada , 1985, IEEE Software.

[20]  Brian Marick,et al.  The craft of software testing , 1994 .

[21]  Luciano Baresi,et al.  An Introduction to Software Testing , 2006, FoVMT.

[22]  Jeff Yu Lei,et al.  IPOG/IPOG‐D: efficient test generation for multi‐way combinatorial testing , 2008, Softw. Test. Verification Reliab..

[23]  Yu Lei,et al.  Random vs. Combinatorial Methods for Discrete Event Simulation of a Grid Computer Network , 2009 .

[24]  Patrick J. Schroeder,et al.  Comparing the fault detection effectiveness of n-way and random test suites , 2004, Proceedings. 2004 International Symposium on Empirical Software Engineering, 2004. ISESE '04..

[25]  Charles J. Colbourn,et al.  The density algorithm for pairwise interaction testing , 2007, Softw. Test. Verification Reliab..

[26]  D. Richard Kuhn Fault classes and error detection capability of specification-based testing , 1999, TSEM.

[27]  Patrick J. Schroeder,et al.  Pairwise Testing : A Best Practice That Isn ’ , 2005 .

[28]  D. Richard Kuhn,et al.  FAILURE MODES IN MEDICAL DEVICE SOFTWARE: AN ANALYSIS OF 15 YEARS OF RECALL DATA , 2001 .

[29]  D. Richard Kuhn,et al.  Combinatorial and Random Testing Effectiveness for a Grid Computer Simulator , 2009 .

[30]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[31]  Atif M. Memon,et al.  Using GUI Run-Time State as Feedback to Generate Test Cases , 2007, 29th International Conference on Software Engineering (ICSE'07).

[32]  G. G. Stokes "J." , 1890, The New Yale Book of Quotations.

[33]  Vadim Okun,et al.  Issues in Software Testing with Model Checkers , 2003 .

[34]  D. Richard Kuhn,et al.  Software fault interactions and implications for software testing , 2004, IEEE Transactions on Software Engineering.

[35]  R. L. Erickson,et al.  Improved quality of protocol testing through techniques of experimental design , 1994, Proceedings of ICC/SUPERCOMM'94 - 1994 International Conference on Communications.

[36]  Barry W. Boehm,et al.  Software Engineering Economics , 1993, IEEE Transactions on Software Engineering.

[37]  D. Richard Kuhn,et al.  COMBINATORIAL TESTING , 2011 .

[38]  Nachiappan Nagappan,et al.  Assessing the Relationship between Software Assertions and Faults: An Empirical Investigation , 2006, 2006 17th International Symposium on Software Reliability Engineering.

[39]  C. A. R. Hoare Assertions: A Personal Perspective , 2002, Software Pioneers.

[40]  Jeffrey M. Voas,et al.  Programming with Assertions: A Prospectus , 2004, IT Prof..

[41]  A. Jain,et al.  Workshop on advances in model-based software testing , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[42]  Ajitha Rajan,et al.  Interaction Testing in Model-Based Development: Effect on Model-Coverage , 2006, 2006 13th Asia Pacific Software Engineering Conference (APSEC'06).

[43]  Mladen A. Vouk,et al.  Optimizing effectiveness and efficiency of software testing: a hybrid approach , 2006 .

[44]  Jeff Yu Lei,et al.  Combinatorial Methods for Event Sequence Testing , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[45]  Mladen A. Vouk Software Reliability Engineering , 1999 .

[46]  Jeff Yu Lei,et al.  Combinatorial Software Testing , 2009, Computer.

[47]  Yu Lei,et al.  Automated Combinatorial Test Methods - Beyond Pairwise Testing , 2008 .

[48]  David S. Rosenblum A Practical Approach to Programming With Assertions , 1995, IEEE Trans. Software Eng..

[49]  D.M. Cohen,et al.  The Combinatorial Design Approach to Automatic Test Generation , 1996, IEEE Softw..

[50]  M. J. Reilly,et al.  An investigation of the applicability of design of experiments to software testing , 2002, 27th Annual NASA Goddard/IEEE Software Engineering Workshop, 2002. Proceedings..

[51]  M.A. Vouk,et al.  On effectiveness of pairwise methodology for testing network-centric software , 2005, 2005 International Conference on Information and Communication Technology.

[52]  Atif M. Memon,et al.  Studying the fault-detection effectiveness of GUI test cases for rapidly evolving software , 2005, IEEE Transactions on Software Engineering.

[53]  Dolores R. Wallace,et al.  A Testability-Based Assertion Placement Tool for Object-Oriented Software, , 1998 .

[54]  Robert L. Probert,et al.  A practical strategy for testing pair-wise coverage of network interfaces , 1996, Proceedings of ISSRE '96: 7th International Symposium on Software Reliability Engineering.

[55]  Y. Yesha,et al.  Specification mutation for test generation and analysis , 2004 .

[56]  D. Richard Kuhn,et al.  Pseudo-Exhaustive Testing for Software , 2006, 2006 30th Annual IEEE/NASA Software Engineering Workshop.

[57]  Yves Le Traon,et al.  Model-Based Tests for Access Control Policies , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[58]  Vadim Okun,et al.  Testing with Model Checker: Insuring Fault Visibility , 2002 .

[59]  Sérgio Vale Aguiar Campos,et al.  Symbolic Model Checking , 1993, CAV.

[60]  B. McKay nauty User ’ s Guide ( Version 2 . 4 ) , 1990 .

[61]  簡聰富,et al.  物件導向軟體之架構(Object-Oriented Software Construction)探討 , 1989 .

[62]  Albert L. Baker,et al.  JML: A Notation for Detailed Design , 1999, Behavioral Specifications of Businesses and Systems.