The Effectiveness of T-Way Test Data Generation

This paper reports the results of a study comparing the effectiveness of automatically generated tests constructed using random and t-way combinatorial techniques on safety related industrial code using mutation adequacy criteria. A reference point is provided by hand generated test vectors constructed during development to establish minimum acceptance criteria. The study shows that 2-way testing is not adequate measured by mutants kill rate compared with hand generated test set of similar size, but that higher factor t-way test sets can perform at least as well. To reduce the computation overhead of testing large numbers of vectors over large numbers of mutants a staged optimising approach to applying t-way tests is proposed and evaluated which shows improvements in execution time and final test set size.

[1]  Colin L. Mallows,et al.  Factor-covering designs for testing software , 1998 .

[2]  William F. Clocksin,et al.  Programming in Prolog , 1981, Springer Berlin Heidelberg.

[3]  Jeff Yu Lei,et al.  IPOG: A General Strategy for T-Way Software Testing , 2007, 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems (ECBS'07).

[4]  A. Jefferson Offutt,et al.  Mutation analysis using mutant schemata , 1993, ISSTA '93.

[5]  Robert L. Probert,et al.  A practical strategy for testing pair-wise coverage of network interfaces , 1996, Proceedings of ISSRE '96: 7th International Symposium on Software Reliability Engineering.

[6]  Stuart Reid,et al.  An empirical analysis of equivalence partitioning, boundary value analysis and random testing , 1997, Proceedings Fourth International Software Metrics Symposium.

[7]  Dorothy M. Andrews,et al.  An automated program testing methodology and its implementation , 1981, ICSE '81.

[8]  Steven David Prestwich,et al.  Constraint-Based Approaches to the Covering Test Problem , 2004, CSCLP.

[9]  P. D. Coward,et al.  Symbolic execution and testing , 1990 .

[10]  M. Ellims,et al.  The Csaw C Mutation Tool: Initial Results , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[11]  James H. Fetzer.  Program verification: the very idea , 1988, CACM.

[12]  Gary McGraw,et al.  Genetic algorithms for dynamic test data generation , 1997, Proceedings 12th IEEE International Conference Automated Software Engineering.

[13]  John B. Goodenough,et al.  Toward a theory of test data selection , 1975 .

[14]  Hans-Gerhard Gross,et al.  Evolutionary algorithms for the verification of execution time bounds for real-time software , 1999 .

[15]  John A. Clark,et al.  An automated framework for structural test-data generation , 1998, Proceedings 13th IEEE International Conference on Automated Software Engineering (Cat. No.98EX239).

[16]  J. Parelius,et al.  Efficacy of AETG ∗ Test Cases as measured by code coverage , 1995 .

[17]  John A. Clark,et al.  Automated test‐data generation for exception conditions , 2000 .

[18]  Borislav Nikolik.  Solving Constraints Involving Indexed Variables , 1997 .

[19]  Gary McGraw,et al.  Software fault injection: inoculating programs against errors , 1997 .

[20]  J.H. Andrews,et al.  Is mutation an appropriate tool for testing experiments? [software testing] , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[21]  L. Cooper,et al.  Sequential Search: A Method for Solving Constrained Optimization Problems , 1965, JACM.

[22]  Luca Maria Gambardella,et al.  Ant colony system: a cooperative learning approach to the traveling salesman problem , 1997, IEEE Trans. Evol. Comput..

[23]  Leonardo Bottaci.  Instrumenting Programs With Flag Variables For Test Data Search By Genetic Algorithms , 2002, GECCO.

[24]  Robert L. Probert,et al.  A measure for component interaction test coverage , 2001, Proceedings ACS/IEEE International Conference on Computer Systems and Applications.

[25]  Ralf Kneuper.  Limits of formal methods , 2005, Formal Aspects of Computing.

[26]  Pascale Thévenod-Fosse,et al.  Software error analysis: a real case study involving real faults and mutations , 1996, ISSTA '96.

[27]  Anneliese Amschler Andrews,et al.  SADAT - An Automated Testing Tool , 1980, IEEE Trans. Software Eng..

[28]  Roy P. Pargas,et al.  Test‐data generation using genetic algorithms , 1999 .

[29]  Timo Mantere,et al.  Genetic Algorithm Based Software Testing , 1997, ICANNGA.

[30]  Daniel Hoffman,et al.  A framework for table driven testing of Java classes , 2002, Softw. Pract. Exp..

[31]  Gregg Rothermel,et al.  An experimental determination of sufficient mutant operators , 1996, TSEM.

[32]  A. Jefferson Offutt,et al.  Constraint-Based Automatic Test Data Generation , 1991, IEEE Trans. Software Eng..

[33]  Andris Zarins,et al.  SMOTL - A System to Construct Samples for Data Processing Program Debugging , 1979, IEEE Trans. Software Eng..

[34]  Richard G. Hamlet.  Introduction to special section on software testing , 1988, CACM.

[35]  Martyn A. Ould,et al.  Testing-a challenge to method and tool developers , 1991, Softw. Eng. J..

[36]  Richard G. Hamlet,et al.  Partition Testing Does Not Inspire Confidence , 1990, IEEE Trans. Software Eng..

[37]  C. V. Ramamoorthy,et al.  On the Automated Generation of Program Test Data , 1976, IEEE Transactions on Software Engineering.

[38]  Charles J. Colbourn,et al.  One-test-at-a-time heuristic search for interaction test suites , 2007, GECCO '07.

[39]  Yu Lei,et al.  IPOG-IPOG-D: efficient test generation for multi-way combinatorial testing , 2008 .

[40]  Koushik Sen,et al.  DART: directed automated random testing , 2005, PLDI '05.

[41]  Dr. Zbigniew Michalewicz,et al.  How to Solve It: Modern Heuristics , 2004 .

[42]  Tsong Yueh Chen,et al.  On the Relationship Between Partition and Random Testing , 1994, IEEE Trans. Software Eng..

[43]  Arnaud Gotlieb.  Exploiting symmetries to test programs , 2003, 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003..

[44]  Darrel C. Ince,et al.  The Automatic Generation of Test Data , 1987, Comput. J..

[45]  Brett Stevens,et al.  Efficient software testing protocols , 1998, CASCON.

[46]  S. Inglis,et al.  Jumble Java Byte Code to Measure the Effectiveness of Unit Tests , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[47]  Alison Watkins,et al.  Evolutionary test data generation: a comparison of fitness functions , 2006, Softw. Pract. Exp..

[48]  Ivars Peterson,et al.  Fatal Defect: Chasing Killer Computer Bugs , 1995 .

[49]  Richard G. Hamlet,et al.  Implementing prototype testing tools , 1995, Softw. Pract. Exp..

[50]  John A. McDermid,et al.  15 – Software development process models , 1991 .

[51]  Joachim Wegener,et al.  Evolutionary Functional Testing of a Vehicle Brake Assistant System , 2005 .

[52]  Steven David Prestwich,et al.  Constraint Models for the Covering Test Problem , 2006, Constraints.

[53]  James H. Cross,et al.  A Rule-Based Software Test Data Generator , 1991, IEEE Trans. Knowl. Data Eng..

[54]  Mark Harman,et al.  VADA: a transformation-based system for variable dependence analysis , 2002, Proceedings. Second IEEE International Workshop on Source Code Analysis and Manipulation.

[55]  Webb Stacy,et al.  Cognitive bias in software engineering , 1995, CACM.

[56]  Lori A. Clarke,et al.  TEAM: a support environment for testing, evaluation, and analysis , 1988, SDE 3.

[57]  P. David Coward,et al.  A review of software testing , 1988 .

[58]  Allen Goldberg,et al.  Applications of feasible path analysis to program testing , 1994, ISSTA '94.

[59]  I. D. Hill,et al.  Generating good pseudo-random numbers , 2006, Comput. Stat. Data Anal..

[60]  Simeon C. Ntafos,et al.  On Some Reliability Estimation Problems in Random and Partition Testing , 1993, IEEE Trans. Software Eng..

[61]  Myra B. Cohen,et al.  Covering Arrays for Efficient Fault Characterization in Complex Configuration Spaces , 2006, IEEE Trans. Software Eng..

[62]  D. Henriksson,et al.  Test case generation for flexible real-time control systems , 2005, 2005 IEEE Conference on Emerging Technologies and Factory Automation.

[63]  Moheb R. Girgis.  An experimental evaluation of a symbolic execution system , 1992, Softw. Eng. J..

[64]  K. N. King,et al.  A Fortran 77 interpreter for mutation analysis , 1987, PLDI 1987.

[65]  Michael Ellims,et al.  On Wheels, Nuts and Software , 2004, SCS.

[66]  John Barnes,et al.  High Integrity Ada: The Spark Approach , 1997 .

[67]  Bogdan Korel,et al.  Automated Software Test Data Generation , 1990, IEEE Trans. Software Eng..

[68]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[69]  Yu Lei,et al.  In-parameter-order: a test generation strategy for pairwise testing , 1998, Proceedings Third IEEE International High-Assurance Systems Engineering Symposium (Cat. No.98EX231).

[70]  John Morris,et al.  Using symbolic execution to guide test generation , 2005, Softw. Test. Verification Reliab..

[71]  Jacek Czerwonka,et al.  Pairwise Testing in Real World , 2006 .

[72]  Eugene Miya,et al.  On "Software engineering" , 1985, SOEN.

[73]  James J. Filliben,et al.  NIST/SEMATECH e-Handbook of Statistical Methods; Chapter 1: Exploratory Data Analysis , 2003 .

[74]  Elaine J. Weyuker,et al.  Automatically Generating Test Data from a Boolean Specification , 1994, IEEE Trans. Software Eng..

[75]  James C. King,et al.  A new approach to program testing , 1974, Programming Methodology.

[76]  André Baresel,et al.  Evolutionary Testing of Flag Conditions , 2003, GECCO.

[77]  A. Jefferson Offutt,et al.  Combination testing strategies: a survey , 2005, Softw. Test. Verification Reliab..

[78]  Nicola Muscettola,et al.  Challenges and Methods in Testing the Remote Agent Planner , 2000, AIPS.

[79]  John A. Clark,et al.  Automated program flaw finding using simulated annealing , 1998, ISSTA '98.

[80]  Gregg Rothermel,et al.  Proceedings of the 2004 ACM SIGSOFT international symposium on Software testing and analysis , 2004 .

[81]  Joachim Wegener,et al.  Evaluation of Different Fitness Functions for the Evolutionary Testing of an Autonomous Parking System , 2004, GECCO.

[82]  Bojan Cukic,et al.  Comparing Partition and Random Testing via Majorization and Schur Functions , 2003, IEEE Trans. Software Eng..

[83]  Joachim Schimpf,et al.  ECLiPSe: A Platform for Constraint Logic Programming , 1997 .

[84]  Patric R. J. Östergård,et al.  Constructing Covering Designs by Simulated Annealing , 1993 .

[85]  Mike Bauer,et al.  Proceedings of the 2001 conference of the Centre for Advanced Studies on Collaborative Research, November 5-7, 2001, Toronto, Ontario, Canada , 2001, CASCON.

[86]  C. L. Mallows,et al.  Applying design of experiments to software testing: experience report , 1997, ICSE '97.

[87]  Bryan F. Jones,et al.  Automatic structural testing using genetic algorithms , 1996, Softw. Eng. J..

[88]  Thomas Stützle,et al.  GECCO 2007: Genetic and Evolutionary Computation Conference , 2007 .

[89]  Joseph Robert Horgan,et al.  Effect of test set size and block coverage on the fault detection effectiveness , 1994, Proceedings of 1994 IEEE International Symposium on Software Reliability Engineering.

[90]  A. Jefferson Offutt,et al.  Experimental results from an automatic test case generator , 1993, TSEM.

[91]  Mary Lou Soffa,et al.  Generating test data for branch coverage , 2000, Proceedings ASE 2000. Fifteenth IEEE International Conference on Automated Software Engineering.

[92]  Gary McGraw,et al.  Generating Software Test Data by Evolution , 2001, IEEE Trans. Software Eng..

[93]  Christophe Meudec,et al.  Automatic Test Data Generation from Embedded C Code , 2004, SAFECOMP.

[94]  Michael L. Fredman,et al.  The AETG System: An Approach to Testing Based on Combinatiorial Design , 1997, IEEE Trans. Software Eng..

[95]  Carlos Urias Munoz,et al.  Automatic Generation of Random Self-Checking Test Cases , 1983, IBM Syst. J..

[96]  A. Jefferson Offutt,et al.  The Dynamic Domain Reduction Procedure for Test Data Generation: Design and Algorithms , 1994 .

[97]  Lee J. White,et al.  A Domain Strategy for Computer Program Testing , 1980, IEEE Transactions on Software Engineering.

[98]  D.M. Cohen,et al.  The Combinatorial Design Approach to Automatic Test Generation , 1996, IEEE Softw..

[99]  Johan Karlsson,et al.  Experimental dependability evaluation of a fail-bounded jet engine control system for unmanned aerial vehicles , 2005, 2005 International Conference on Dependable Systems and Networks (DSN'05).

[100]  A. Jefferson Offutt,et al.  An evaluation of combination strategies for test case selection , 2006, Empirical Software Engineering.

[101]  Mario Jino,et al.  Automatic Test Data Generation for Program Paths Using Genetic Algorithms , 2002, Int. J. Softw. Eng. Knowl. Eng..

[102]  Robert Brownlie,et al.  Robust testing of AT&T PMX/StarMAIL using OATS , 1992, AT&T Technical Journal.

[103]  Robert L. Probert,et al.  Formulation of the Interaction Test Coverage Problem as an Integer Program , 2002, TestCom.

[104]  David F. McAllister,et al.  An experimental evaluation of the effectiveness of random testing of fault-tolerant software , 1986 .

[105]  K. N. King,et al.  A fortran language system for mutation‐based software testing , 1991, Softw. Pract. Exp..

[106]  Dave Towey,et al.  Restricted Random Testing , 2002, ECSQ.

[107]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[108]  Christophe Meudec.  ATGen: automatic test data generation using constraint logic programming and symbolic execution † , 2001, Softw. Test. Verification Reliab..

[109]  Keith Jackson,et al.  IS0 9001: Making the Right Mistakes , 2000 .

[110]  Walter F. Tichy,et al.  Should Computer Scientists Experiment More? , 1998, Computer.

[111]  Jiangfan Shi,et al.  Exploiting Constraint Solving History to Construct Interaction Test Suites , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[112]  Yoonsik Cheon,et al.  A Fitness Function for Modular Evolutionary Testing of Object-Oriented Programs , 2005 .

[113]  Bryan F. Jones,et al.  A Strategy for Using Genetic Algorithms to Automate Branch and Fault-Based Testing , 1998, Comput. J..

[114]  Peter G. Neumann,et al.  Computer-related risks , 1994 .

[115]  A. Jefferson Offutt,et al.  Introduction to Software Testing , 2008 .

[116]  Jacques Cohen,et al.  Constraint logic programming languages , 1990, CACM.

[117]  Eugenia Díaz,et al.  Automated software testing using a metaheuristic technique based on Tabu search , 2003, 18th IEEE International Conference on Automated Software Engineering, 2003. Proceedings..

[118]  Glenford J. Myers,et al.  A controlled experiment in program testing and code walkthroughs/inspections , 1978, CACM.

[119]  A. E. Eiben,et al.  Introduction to Evolutionary Computing , 2003, Natural Computing Series.

[120]  Yu-Wen Tung,et al.  Automating test case generation for the new generation mission software system , 2000, 2000 IEEE Aerospace Conference. Proceedings (Cat. No.00TH8484).

[121]  Leonardo Bottaci,et al.  Predicate Expression Cost Functions to Guide Evolutionary Search for Test Data , 2003, GECCO.

[122]  Ron Patton,et al.  Software Testing , 2000 .

[123]  Daniel P. Siewiorek,et al.  A dimensionality model approach to testing and improving software robustness , 1999, 1999 IEEE AUTOTESTCON Proceedings (Cat. No.99CH36323).

[124]  Yashwant K. Malaiya,et al.  Automatic test generation using checkpoint encoding and antirandom testing , 1997, Proceedings The Eighth International Symposium on Software Reliability Engineering.

[125]  Phyllis G. Frankl,et al.  Comparison of delivered reliability of branch, data flow and operational testing: A case study , 2000, ISSTA '00.

[126]  Bev Littlewood,et al.  Evaluating Testing Methods by Delivered Reliability , 1998, IEEE Trans. Software Eng..

[127]  Yves Ledru,et al.  Directed random reduction of combinatorial test suites , 2007, RT '07.

[128]  Numerical Algorithms Group , 2002 .

[129]  P. David Coward,et al.  The symbolic execution of software - the SYM-BOL system , 1994, Chapman and Hall computer science : research and practice.

[130]  Lee J. White,et al.  Complexity of testing iterated borders for structured programs , 1988, [1988] Proceedings. Second Workshop on Software Testing, Verification, and Analysis.

[131]  Jin-Cherng Lin,et al.  Automatic test data generation for path testing using GAs , 2001, Inf. Sci..

[132]  Myra B. Cohen,et al.  A deterministic density algorithm for pairwise interaction coverage , 2004, IASTED Conf. on Software Engineering.

[133]  Douglas C. Montgomery,et al.  Software performance testing using covering arrays: efficient screening designs with categorical factors , 2005, WOSP '05.

[134]  Elaine J. Weyuker,et al.  An Applicable Family of Data Flow Testing Criteria , 1988, IEEE Trans. Software Eng..

[135]  Wei-Tek Tsai,et al.  Progressive ranking and composition of Web services using covering arrays , 2005, 10th IEEE International Workshop on Object-Oriented Real-Time Dependable Systems.

[136]  John J. Grefenstette,et al.  Test and evaluation by genetic algorithms , 1993, IEEE Expert.

[137]  Keith E. Williamson,et al.  Test data generation and feasible path analysis , 1994, ISSTA '94.

[138]  Edward J. McCluskey,et al.  Pseudorandom Testing , 1987, IEEE Transactions on Computers.

[139]  V. Lakshmi Narasimhan,et al.  ADTEST: A Test Data Generation Suite for Ada Software Systems , 1997, IEEE Trans. Software Eng..

[140]  Lee Copeland,et al.  A Practitioner's Guide to Software Test Design , 2003 .

[141]  Yves Crouzet,et al.  An experimental study on software structural testing: deterministic versus random input generation , 1991, [1991] Digest of Papers. Fault-Tolerant Computing: The Twenty-First International Symposium.

[142]  Mary Lou Soffa,et al.  Automated test data generation using an iterative relaxation method , 1998, SIGSOFT '98/FSE-6.

[143]  Renée C. Bryce,et al.  A framework of greedy methods for constructing interaction test suites , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[144]  Phil McMinn,et al.  Search‐based software test data generation: a survey , 2004, Softw. Test. Verification Reliab..

[145]  Wynne Hsu,et al.  DESIGN OF MUTANT OPERATORS FOR THE C PROGRAMMING LANGUAGE , 2006 .

[146]  D. Richard Kuhn,et al.  FAILURE MODES IN MEDICAL DEVICE SOFTWARE: AN ANALYSIS OF 15 YEARS OF RECALL DATA , 2001 .

[147]  Huifang Yin.  Antirandom Test Patterns Generation Tool , 1996 .

[148]  Elaine J. Weyuker,et al.  The evaluation of program-based software test data adequacy criteria , 1988, CACM.

[149]  Phil McMinn,et al.  Evolutionary testing of state-based programs , 2005, GECCO '05.

[150]  Elaine J. Weyuker,et al.  On Testing Non-Testable Programs , 1982, Comput. J..

[151]  Daniel Hoffman,et al.  Module test case generation , 1989 .

[152]  David Clark,et al.  Safety and Security Analysis of Object-Oriented Models , 2002, SAFECOMP.

[153]  Edward J. McCluskey,et al.  Concurrent Error Detection Using Watchdog Processors - A Survey , 1988, IEEE Trans. Computers.

[154]  Oliver Laitenberger,et al.  Studying the effects of code inspection and structural testing on software quality , 1998, Proceedings Ninth International Symposium on Software Reliability Engineering (Cat. No.98TB100257).

[155]  Ray A. Jarvis,et al.  Adaptive Global Search in a Time-Variant Environment Using a Probabilistic Automaton with Pattern Recognition Supervision , 1970, IEEE Trans. Syst. Sci. Cybern..

[156]  Lori A. Clarke,et al.  A System to Generate Test Data and Symbolically Execute Programs , 1976, IEEE Transactions on Software Engineering.

[157]  W. Eric Wong,et al.  Effect of test set minimization on fault detection effectiveness , 1998 .

[158]  M. Petre,et al.  AETG vs . Man : an Assessment of the Effectiveness of Combinatorial Test Data Generation , 2007 .

[159]  A. J. Offutt.  A practical system for mutation testing: help for the common programmer , 1994, Proceedings., International Test Conference.

[160]  John A. Clark,et al.  Towards industrially applicable formal methods: three small steps, and one giant leap , 1998, Proceedings Second International Conference on Formal Engineering Methods (Cat.No.98EX241).

[161]  Myra B. Cohen,et al.  Augmenting simulated annealing to build interaction test suites , 2003, 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003..

[162]  Simeon C. Ntafos,et al.  An Evaluation of Random Testing , 1984, IEEE Transactions on Software Engineering.

[163]  Hoyt Lougee,et al.  SOFTWARE CONSIDERATIONS IN AIRBORNE SYSTEMS AND EQUIPMENT CERTIFICATION , 2001 .

[164]  André Baresel,et al.  Structural and Functional Sequence Test of Dynamic and State-Based Software with Evolutionary Algorithms , 2003, GECCO.

[165]  Bin Wang,et al.  A unified fitness function calculation rule for flag conditions to improve evolutionary testing , 2005, ASE '05.

[166]  Paul Lukowicz,et al.  Experimental evaluation in computer science: A quantitative study , 1995, J. Syst. Softw..

[167]  Jürgen Ackermann,et al.  Robust control prevents car skidding , 1997 .

[168]  Richard G. Hamlet,et al.  Testing Programs with the Aid of a Compiler , 1977, IEEE Transactions on Software Engineering.

[169]  Tsong Yueh Chen,et al.  Proportional sampling strategy: a compendium and some insights , 2001, J. Syst. Softw..

[170]  Mark Harman,et al.  Testability Transformation for Efficient Automated Test Data Search in the Presence of Nesting , 2005 .

[171]  David Notkin,et al.  Symstra: A Framework for Generating Object-Oriented Unit Tests Using Symbolic Execution , 2005, TACAS.

[172]  Boris Beizer,et al.  Software testing techniques (2. ed.) , 1990 .

[173]  Richard A. Kemmerer,et al.  Unisex: A unix‐based symbolic executor for pascal , 1985, Softw. Pract. Exp..

[174]  R. Bell,et al.  IEC 61508: functional safety of electrical/electronic/ programme electronic safety-related systems: overview , 1999 .

[175]  Tatsuhiro Tsuchiya,et al.  A new method for constructing pair-wise covering designs for software testing , 2002, Information Processing Letters.

[176]  Rolf Isermann,et al.  Fault-tolerant drive-by-wire systems , 2002 .

[177]  A. Offutt.  The coupling effect: fact or fiction , 1989 .

[178]  Joachim Wegener,et al.  Evolutionary test environment for automatic structural testing , 2001, Inf. Softw. Technol..

[179]  Victor R. Basili,et al.  Comparing the Effectiveness of Software Testing Strategies , 1987, IEEE Transactions on Software Engineering.

[180]  Myra B. Cohen,et al.  Constructing test suites for interaction testing , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[181]  Lee J. White,et al.  Path testing of computer programs with loops using a tool for simple loop patterns , 1991, Softw. Pract. Exp..

[182]  Daniel Ashlock,et al.  Finding Designs with Genetic Algorithms , 1996 .

[183]  Stephen Parker,et al.  Design and Analysis of a Robust Real-Time Engine Control Network , 2002, IEEE Micro.

[184]  John A. Clark,et al.  The Way Forward for Unifying Dynamic Test Case Generation: The Optimisation-based Approach , 1998 .

[185]  Myra B. Cohen,et al.  Interaction testing of highly-configurable systems in the presence of constraints , 2007, ISSTA '07.

[186]  Michael Ellims,et al.  The Economics of Unit Testing , 2006, Empirical Software Engineering.

[187]  Baowen Xu,et al.  Applying combinatorial method to test browser compatibility , 2003, Fifth International Symposium on Multimedia Software Engineering, 2003. Proceedings..

[188]  Jun Yan,et al.  Backtracking Algorithms and Search Heuristics to Generate Test Suites for Combinatorial Testing , 2006, 30th Annual International Computer Software and Applications Conference (COMPSAC'06).

[189]  John A. McDermid,et al.  Software in safety critical systems?achievement and prediction , 2006 .

[190]  Tsong Yueh Chen,et al.  On the Expected Number of Failures Detected by Subdomain Testing and Random Testing , 1996, IEEE Trans. Software Eng..

[191]  Ashish Jain,et al.  Model-based testing of a highly programmable system , 1998, Proceedings Ninth International Symposium on Software Reliability Engineering (Cat. No.98TB100257).

[192]  Pascal Van Hentenryck,et al.  The Constraint Logic Programming Language CHIP , 1988, FGCS.

[193]  Roy S. Freedman,et al.  Testability of Software Components , 1991, IEEE Trans. Software Eng..

[194]  G. Keppel,et al.  Introduction to Design and Analysis : A Student's Handbook , 1980 .

[195]  P. Cameron.  Combinatorics: Topics, Techniques, Algorithms , 1995 .

[196]  Michael Ellims,et al.  Unit Testing Techniques and Tool Support , 1999 .

[197]  Donald J. Berndt,et al.  Breeding software test cases for complex systems , 2004, 37th Annual Hawaii International Conference on System Sciences, 2004. Proceedings of the.

[198]  A Jeeerson Ooutt,et al.  Subsumption of Condition Coverage Techniques by Mutation Testing , 1996 .

[199]  Hans-Gerhard Groß,et al.  Search-based execution-time verification in object-oriented and component-based real-time system development , 2003, Proceedings of the Eighth International Workshop on Object-Oriented Real-Time Dependable Systems, 2003. (WORDS 2003)..

[200]  Peter Naur,et al.  Programming by action clusters , 1969 .

[201]  Jim Fowler,et al.  Practical Statistics for Field Biology , 1991 .

[202]  P. Thevenod-Fosse,et al.  From random testing of hardware to statistical testing of software , 1991, [1991] Proceedings, Advanced Computer Technology, Reliable Systems and Applications.

[203]  Patrick J. Schroeder,et al.  Comparing the fault detection effectiveness of n-way and random test suites , 2004, Proceedings. 2004 International Symposium on Empirical Software Engineering, 2004. ISESE '04..

[204]  William E. Howden,et al.  Methodology for the Generation of Program Test Data , 1975, IEEE Transactions on Computers.

[205]  André Baresel,et al.  Fitness Function Design To Improve Evolutionary Structural Testing , 2002, GECCO.

[206]  Mirko Conrad,et al.  Evolutionary safety testing of embedded control software by automatically generating compact test data sequences , 2005 .

[207]  Jeff Yu Lei,et al.  Practical Combinatorial Testing: Beyond Pairwise , 2008, IT Professional.

[208]  William E. Howden.  DISSECT—A Symbolic Evaluation and Program Testing System , 1978, IEEE Transactions on Software Engineering.

[209]  Marc J. Balcer,et al.  The category-partition method for specifying and generating fuctional tests , 1988, CACM.

[210]  Alain Denise,et al.  A new way of automating statistical testing methods , 2001, Proceedings 16th Annual International Conference on Automated Software Engineering (ASE 2001).

[211]  Gregg Rothermel,et al.  An empirical study of the effects of minimization on the fault detection capabilities of test suites , 1998, Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272).

[212]  Matthias Grochtmann,et al.  Testing Temporal Correctness of Real-Time Systems by Means of Genetic Algorithms , 1997 .

[213]  Martin L. Shooman.  Avionics software problem occurrence rates , 1996, Proceedings of ISSRE '96: 7th International Symposium on Software Reliability Engineering.

[214]  Hermann Kopetz.  Automotive electronics , 1999, Proceedings of 11th Euromicro Conference on Real-Time Systems. Euromicro RTS'99.

[215]  Jean-Marc Jézéquel,et al.  Building trust into OO components using a genetic analogy , 2000, Proceedings 11th International Symposium on Software Reliability Engineering. ISSRE 2000.

[216]  Bogdan Korel,et al.  The chaining approach for software test data generation , 1996, TSEM.

[217]  Martin R. Woodward,et al.  Experience with Path Analysis and Testing of Programs , 1980, IEEE Transactions on Software Engineering.

[218]  Cem Kaner,et al.  Lessons learned in software testing ; a context - driven approach , 2002 .

[219]  Tatsuhiro Tsuchiya,et al.  Non-specification-based approaches to logic testing for software , 2002, Inf. Softw. Technol..

[220]  Isolde Wildgruber,et al.  The Real-Time Behavior of the Steam Boiler , 1995, Formal Methods for Industrial Applications.

[221]  Siddhartha R. Dalal,et al.  Model-based testing in practice , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[222]  David Goldberg,et al.  What every computer scientist should know about floating-point arithmetic , 1991, CSUR.

[223]  William E. Howden,et al.  An evaluation of the effectiveness of symbolic testing , 1978, Softw. Pract. Exp..

[224]  Simeon C. Ntafos,et al.  On Comparisons of Random, Partition, and Proportional Partition Testing , 2001, IEEE Trans. Software Eng..

[225]  Antonia Bertolino,et al.  Automatic Generation of Path Covers Based on the Control Flow Analysis of Computer Programs , 1994, IEEE Trans. Software Eng..

[226]  Mark Harman,et al.  Evolutionary testing in the presence of loop-assigned flags: a testability transformation approach , 2004, ISSTA '04.

[227]  Hans-Gerhard Groß,et al.  Evolutionary Testing In Component-based Real-time System Construction , 2002, GECCO.

[228]  John A. Clark,et al.  Search-based mutation testing for Simulink models , 2005, GECCO '05.

[229]  James C. King,et al.  Applications of Symbolic Execution to Program Testing , 1978, Computer.

[230]  George Mason,et al.  Procedures for Reducing the Size of Coverage-based Test Sets , 1995 .

[231]  John A. Clark,et al.  A search-based automated test-data generation framework for safety-critical systems , 2002 .

[232]  G. T. Daich.  Testing combinations of parameters made easy [software testing] , 2003, Proceedings AUTOTESTCON 2003. IEEE Systems Readiness Technology Conference..

[233]  Steven P. Miller,et al.  Applicability of modified condition/decision coverage to software testing , 1994, Softw. Eng. J..

[234]  Thomas J. Ostrand,et al.  Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria , 1994, Proceedings of 16th International Conference on Software Engineering.

[235]  Charles J. Colbourn,et al.  Prioritized interaction testing for pair-wise coverage with seeding and constraints , 2006, Inf. Softw. Technol..

[236]  William E. Howden,et al.  Reliability of the Path Analysis Testing Strategy , 1976, IEEE Transactions on Software Engineering.

[237]  Richard G. Hamlet.  Are we testing for true reliability? , 1992, IEEE Software.

[238]  M. J. Reilly,et al.  An investigation of the applicability of design of experiments to software testing , 2002, 27th Annual NASA Goddard/IEEE Software Engineering Workshop, 2002. Proceedings..

[239]  Matthias Grochtmann.  Evolutionary Testing of Temporal Correctness , 1998 .

[240]  Jim Rutherford,et al.  Practical Experiment Designs for Engineers and Scientists , 2002, Technometrics.

[241]  Michael Ellims,et al.  Frameworks for Power and Systems Management in Hybrid Vehicles: Challenges and Prospects , 2006 .

[242]  Julian F. Miller,et al.  Genetic and Evolutionary Computation — GECCO 2003 , 2003, Lecture Notes in Computer Science.

[243]  Mary Jean Harrold,et al.  Test-Suite Reduction and Prioritization for Modified Condition/Decision Coverage , 2003, IEEE Trans. Software Eng..

[244]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[245]  A. Jefferson Offutt,et al.  An integrated automatic test data generation system , 1991, J. Syst. Integr..

[246]  Eugenia Díaz,et al.  A scatter search approach for automated branch coverage in software testing , 2007 .

[247]  Michael Ellims.  Is Security Necessary for Safety , 2006 .

[248]  Mary Lou Soffa,et al.  UNA based iterative test data generation and its evaluation , 1999, 14th IEEE International Conference on Automated Software Engineering.

[249]  Paul Ammann,et al.  Using formal methods to derive test frames in category-partition testing , 1994, Proceedings of COMPASS'94 - 1994 IEEE 9th Annual Conference on Computer Assurance.

[250]  William E. Howden,et al.  Functional Program Testing , 1978, IEEE Transactions on Software Engineering.

[251]  Leonard Steinborn,et al.  International Organization for Standardization ISO 9001:1994 Quality Systems — Model for Quality Assurance in Design, Development, Production, Installation and Servicing , 2004 .

[252]  Phyllis G. Frankl,et al.  Further empirical studies of test effectiveness , 1998, SIGSOFT '98/FSE-6.

[253]  Yoichi Hayashi,et al.  Neural expert system using fuzzy teaching input and its application to medical diagnosis , 1994 .

[254]  David L. Spooner,et al.  Automatic Generation of Floating-Point Test Data , 1976, IEEE Transactions on Software Engineering.

[255]  S. K. Park,et al.  Random number generators: good ones are hard to find , 1988, CACM.

[256]  Yashwant K. Malaiya,et al.  Antirandom testing: getting the most out of black-box testing , 1995, Proceedings of Sixth International Symposium on Software Reliability Engineering. ISSRE'95.

[257]  D. Richard Kuhn,et al.  Software fault interactions and implications for software testing , 2004, IEEE Transactions on Software Engineering.

[258]  Krzysztof R. Apt,et al.  Principles of constraint programming , 2003 .

[259]  Oliver Buehler,et al.  Evolutionary Functional Testing of an Automated Parking System , 2003 .

[260]  Joachim Wegener,et al.  A Comparison of Static Analysis and Evolutionary Testing for the Verification of Timing Constraints , 1998, Proceedings. Fourth IEEE Real-Time Technology and Applications Symposium (Cat. No.98TB100245).

[261]  Praful V. Bhansali.  The MCDC paradox , 2007, SOEN.

[262]  Mark Harman,et al.  Testability transformation , 2004, IEEE Transactions on Software Engineering.

[263]  Egon Börger,et al.  Formal methods for industrial applications : specifying and programming the steam boiler control , 1996 .

[264]  Mark Harman,et al.  Improving Evolutionary Testing By Flag Removal , 2002, GECCO.

[265]  Charles J. Colbourn,et al.  Test prioritization for pairwise interaction coverage , 2005, A-MOST.

[266]  C. A. R. Hoare,et al.  How Did Software Get So Reliable Without Proof? , 1996, FME.

[267]  Gregg Rothermel,et al.  Test case prioritization: an empirical study , 1999, Proceedings IEEE International Conference on Software Maintenance - 1999 (ICSM'99). 'Software Maintenance for Business Change' (Cat. No.99CB36360).

[268]  Hélène Waeselynck,et al.  Property-oriented testing: a strategy for exploring dangerous scenarios , 2003, SAC '03.

[269]  Jeremy M. Harrell.  Orthogonal Array Testing Strategy (OATS) Technique , 2001 .

[270]  Yu Lei,et al.  A Test Generation Strategy for Pairwise Testing , 2002, IEEE Trans. Software Eng..

[271]  D. Richard Kuhn,et al.  Pseudo-Exhaustive Testing for Software , 2006, 2006 30th Annual IEEE/NASA Software Engineering Workshop.

[272]  Micah Clark,et al.  Validation and verification of the remote agent for spacecraft autonomy , 1999, 1999 IEEE Aerospace Conference. Proceedings (Cat. No.99TH8403).

[273]  A. Jefferson Offutt,et al.  Investigations of the software testing coupling effect , 1992, TSEM.

[274]  Jerry Huller.  Reducing Time to Market with Combinatorial Design Method Testing , 2002 .

[275]  Boundary values and automated component testing , 1999 .

[276]  Yves Le Traon,et al.  Testing-for-trust: the genetic selection model applied to component qualification , 2000, Proceedings 33rd International Conference on Technology of Object-Oriented Languages and Systems TOOLS 33.

[277]  D. W. Cooper.  Adaptive testing , 1976, ICSE '76.

[278]  Gregory T. Daich.  New Spreadsheet Tool Helps Determine Minimal Set of Test Parameter Combinations , 2003 .

[279]  Paul Piwowarski,et al.  Coverage measurement experience during function test , 1993, Proceedings of 1993 15th International Conference on Software Engineering.

[280]  Jean-Marc Jézéquel,et al.  Genes and bacteria for automatic test cases optimization in the .NET environment , 2002, 13th International Symposium on Software Reliability Engineering, 2002. Proceedings..

[281]  Charles J. Colbourn,et al.  Experimental designs in software engineering: d-optimal designs and covering arrays , 2004, WISER '04.

[282]  Phyllis G. Frankl,et al.  All-uses vs mutation testing: An experimental comparison of effectiveness , 1997, J. Syst. Softw..

[283]  Phyllis G. Frankl,et al.  An Experimental Comparison of the Effectiveness of Branch Testing and Data Flow Testing , 1993, IEEE Trans. Software Eng..

[284]  William H. Press,et al.  Numerical recipes in C. The art of scientific computing , 1987 .

[285]  Walter J. Gutjahr,et al.  Partition Testing vs. Random Testing: The Influence of Uncertainty , 1999, IEEE Trans. Software Eng..

[286]  Phil McMinn,et al.  Hybridizing Evolutionary Testing with the Chaining Approach , 2004, GECCO.

[287]  Pankaj Mudholkar,et al.  Software Testing , 2002, Computer.

[288]  Siddhartha R. Dalal,et al.  The Automatic Efficient Test Generator (AETG) system , 1994, Proceedings of 1994 IEEE International Symposium on Software Reliability Engineering.

[289]  Phil McMinn,et al.  The State Problem for Evolutionary Testing , 2003, GECCO.

[290]  Elaine J. Weyuker,et al.  Analyzing Partition Testing Strategies , 1991, IEEE Trans. Software Eng..

[291]  Giuliano Antoniol,et al.  Automatic unit test data generation using mixed-integer linear programming and execution trees , 1999, Proceedings IEEE International Conference on Software Maintenance - 1999 (ICSM'99). 'Software Maintenance for Business Change' (Cat. No.99CB36360).

[292]  Timothy E. Lindquist,et al.  Test-case generation with IOGen , 1988, IEEE Software.

[293]  Hélène Waeselynck,et al.  STATEMATE applied to statistical software testing , 1993, ISSTA '93.

[294]  Robert Mandl,et al.  Orthogonal Latin squares: an application of experiment design to compiler testing , 1985, CACM.

[295]  Anneliese Amschler Andrews,et al.  Fast antirandom (FAR) test generation , 1998, Proceedings Third IEEE International High-Assurance Systems Engineering Symposium (Cat. No.98EX231).

[296]  Anura P. Jayasumana,et al.  Antirandom vs. pseudorandom testing , 1998, Proceedings International Conference on Computer Design. VLSI in Computers and Processors (Cat. No.98CB36273).

[297]  Erik Kamsties,et al.  An Empirical Evaluation of Three Defect-Detection Techniques , 1995, ESEC.

[298]  Vijayan N. Nair,et al.  A statistical assessment of some software testing strategies and application of experimental design techniques , 1998 .

[299]  Mark N. Wegman,et al.  Efficiently computing static single assignment form and the control dependence graph , 1991, TOPL.

[300]  Myra B. Cohen,et al.  Constructing Interaction Test Suites for Highly-Configurable Systems in the Presence of Constraints: A Greedy Approach , 2008, IEEE Transactions on Software Engineering.

[301]  Walter F. Tichy,et al.  Proceedings 25th International Conference on Software Engineering , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[302]  Jon Edvardsson,et al.  A Survey on Automatic Test Data Generation , 2002 .

[303]  Alan W. Williams,et al.  Determination of Test Configurations for Pair-Wise Interaction Coverage , 2000, TestCom.

[304]  Koen Claessen,et al.  QuickCheck: a lightweight tool for random testing of Haskell programs , 2000, ICFP.

[305]  Marc Roper.  Computer aided software testing using genetic algorithms , 1997 .

[306]  Myra B. Cohen,et al.  A variable strength interaction testing of components , 2003, Proceedings 27th Annual International Computer Software and Applications Conference. COMPAC 2003.

[307]  K. Burr,et al.  Combinatorial Test Techniques : Table-based Automation , Test Generation and Code Coverage , 1998 .

[308]  W. B. Perkinson.  A methodology for designing and executing ISDN feature tests, using automated test systems , 1992, [Conference Record] GLOBECOM '92 - Communications for Global Users: IEEE.

[309]  Pascal Van Hentenryck,et al.  Constraint Satisfaction Using Constraint Logic Programming , 1992, Artif. Intell..

[310]  Joachim Wegener,et al.  Evaluating Evolutionary Testability with Software-Measurements , 2004, GECCO.

[311]  Douglas C. Schmidt,et al.  Skoll: distributed continuous quality assurance , 2004, Proceedings. 26th International Conference on Software Engineering.

[312]  Gary McGraw,et al.  Automated software test data generation for complex programs , 1998, Proceedings 13th IEEE International Conference on Automated Software Engineering (Cat. No.98EX239).

[313]  Joachim Wegener,et al.  Testing real-time systems using genetic algorithms , 1997, Software Quality Journal.

[314]  Tatsuhiro Tsuchiya,et al.  Using artificial life techniques to generate test cases for combinatorial testing , 2004, Proceedings of the 28th Annual International Computer Software and Applications Conference, 2004. COMPSAC 2004..

[315]  Tsong Yueh Chen,et al.  On the test allocations for the best lower bound performance of partition testing , 1998, Proceedings 1998 Australian Software Engineering Conference (Cat. No.98EX233).

[316]  Daniel Hoffman,et al.  Automated Module Testing in Prolog , 1991, IEEE Trans. Software Eng..

[317]  Arnaud Gotlieb,et al.  Automatic test data generation using constraint solving techniques , 1998, ISSTA '98.

[318]  Jeff Yu Lei,et al.  IPOG/IPOG‐D: efficient test generation for multi‐way combinatorial testing , 2008, Softw. Test. Verification Reliab..

[319]  Nancy G. Leveson,et al.  Safeware: System Safety and Computers , 1995 .

[320]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[321]  Karl N. Levitt,et al.  SELECT—a formal system for testing and debugging programs by symbolic execution , 1975 .

[322]  R. L. Erickson,et al.  Improved quality of protocol testing through techniques of experimental design , 1994, Proceedings of ICC/SUPERCOMM'94 - 1994 International Conference on Communications.

[323]  Alberto Pasquini,et al.  Sensitivity of reliability-growth models to operational profile errors vs. testing accuracy [software testing] , 1996, IEEE Trans. Reliab..

[324]  Egon Börger,et al.  Formal Methods for Industrial Applications , 1996, Lecture Notes in Computer Science.

[325]  Bogdan Korel,et al.  Assertion-oriented automated test data generation , 1996, Proceedings of IEEE 18th International Conference on Software Engineering.

[326]  Joel A. Richardson,et al.  Algorithm 454: the complex method for constrained optimization [E4] , 1973 .