An orchestrated survey of methodologies for automated software test case generation

Test case generation is among the most labour-intensive tasks in software testing. It also has a strong impact on the effectiveness and efficiency of software testing. For these reasons, it has been one of the most active research topics in software testing for several decades, resulting in many different approaches and tools. This paper presents an orchestrated survey of the most prominent techniques for automatic generation of software test cases, reviewed in self-standing sections. The techniques presented include: (a) structural testing using symbolic execution, (b) model-based testing, (c) combinatorial testing, (d) random testing and its variant of adaptive random testing, and (e) search-based testing. Each section is contributed by world-renowned active researchers on the technique, and briefly covers the basic ideas underlying the method, the current state of the art, a discussion of the open research problems, and a perspective of the future development of the approach. As a whole, the paper aims at giving an introductory, up-to-date and (relatively) short overview of research in automatic test case generation, while ensuring a comprehensive and authoritative treatment.

[1]  Giuliano Antoniol,et al.  The use of search‐based optimization techniques to schedule and staff software projects: an approach and an empirical study , 2011, Softw. Pract. Exp..

[2]  Phil McMinn,et al.  Search-Based Software Testing: Past, Present and Future , 2011, 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.

[3]  Patrice Godefroid,et al.  Automated Whitebox Fuzz Testing , 2008, NDSS.

[4]  Mark Harman,et al.  Automated web application testing using search based software engineering , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[5]  Myra B. Cohen,et al.  Interaction Coverage Meets Path Coverage by SMT Constraint Solving , 2009, TestCom/FATES.

[6]  Dave Towey,et al.  Forgetting Test Cases , 2006, 30th Annual International Computer Software and Applications Conference (COMPSAC'06).

[7]  Paolo Tonella,et al.  Evolutionary testing of classes , 2004, ISSTA '04.

[8]  Mark Harman,et al.  Reducing qualitative human oracle costs associated with automatically generated test data , 2010, STOV '10.

[9]  Xin Yao,et al.  Coevolving programs and unit tests from their specification , 2007, ASE.

[10]  Mark Harman,et al.  Search-based software engineering , 2001, Inf. Softw. Technol..

[11]  Mike Papadakis,et al.  Automatic Mutation Test Case Generation via Dynamic Symbolic Execution , 2010, 2010 IEEE 21st International Symposium on Software Reliability Engineering.

[12]  Mary Lou Soffa,et al.  TimeAware test suite prioritization , 2006, ISSTA '06.

[13]  Amit M. Paradkar,et al.  Less is More: A Minimalistic Approach to UML Model-Based Conformance Test Generation , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[14]  Margus Veanes,et al.  Optimal strategies for testing nondeterministic systems , 2004, ISSTA '04.

[15]  Jeff Yu Lei,et al.  Combinatorial Methods for Event Sequence Testing , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[16]  Ankur Taly,et al.  Automated synthesis of symbolic instruction encodings from I/O samples , 2012, PLDI.

[17]  Michael Luck,et al.  Evolutionary testing of autonomous software agents , 2009, Autonomous Agents and Multi-Agent Systems.

[18]  Charles J. Colbourn,et al.  One-test-at-a-time heuristic search for interaction test suites , 2007, GECCO '07.

[19]  Kai-Yuan Cai,et al.  Optimal software testing and adaptive software testing in the context of software cybernetics , 2002, Inf. Softw. Technol..

[20]  Yu Lei,et al.  IPOG-IPOG-D: efficient test generation for multi-way combinatorial testing , 2008 .

[21]  Giuliano Antoniol,et al.  Improving network applications security: a new heuristic to generate stress testing data , 2005, GECCO '05.

[22]  Michael Hicks,et al.  Directed Symbolic Execution , 2011, SAS.

[23]  2 Model Based Test Generation Tools , 2022 .

[24]  Sarfraz Khurshid,et al.  Test generation for graphical user interfaces based on symbolic execution , 2008, AST '08.

[25]  Rupak Majumdar,et al.  Directed test generation using symbolic grammars , 2007, ASE.

[26]  Mark Harman,et al.  Software Engineering Meets Evolutionary Computation , 2011, Computer.

[27]  Yuanyuan Zhang,et al.  Search Based Requirements Optimisation: Existing Work and Challenges , 2008, REFSQ.

[28]  Mark Harman,et al.  Clustering test cases to achieve effective and scalable prioritisation incorporating expert knowledge , 2009, ISSTA.

[29]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[30]  Wolfgang Grieskamp,et al.  Multi-paradigmatic Model-Based Testing , 2006, FATES/RV.

[31]  Tsong Yueh Chen,et al.  Mirror adaptive random testing , 2004, Inf. Softw. Technol..

[32]  Brian Nielsen,et al.  Automated Test Generation from Timed Automata , 2001, TACAS.

[33]  C. Colbourn Combinatorial aspects of covering arrays , 2006 .

[34]  Steven David Prestwich,et al.  Constraint Models for the Covering Test Problem , 2006, Constraints.

[35]  Jan Tretmans,et al.  Testing Transition Systems: An Annotated Bibliography , 2000, MOVEP.

[36]  M. P. Vasilevskii Failure diagnosis of automata , 1973 .

[37]  Tsong Yueh Chen,et al.  An upper bound on software testing effectiveness , 2008, TSEM.

[38]  Lee J. White,et al.  A Domain Strategy for Computer Program Testing , 1980, IEEE Transactions on Software Engineering.

[39]  Jeremy Dick,et al.  Automating the Generation and Sequencing of Test Cases from Model-Based Specifications , 1993, FME.

[40]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[41]  Antti Huima,et al.  Implementing Conformiq Qtronic , 2007, TestCom/FATES.

[42]  Ilkka Niemelä,et al.  LCT: An Open Source Concolic Testing Tool for Java Programs , 2011 .

[43]  Elaine J. Weyuker Translatability and Decidability Questions for Restricted Classes of Program Schemas , 1979, SIAM J. Comput..

[44]  Huai Liu,et al.  Adaptive random testing through test profiles , 2011, Softw. Pract. Exp..

[45]  Robert M. Hierons,et al.  Implementation Relations for the Distributed Test Architecture , 2008, TestCom/FATES.

[46]  Myra B. Cohen,et al.  Combinatorial Interaction Regression Testing: A Study of Test Case Generation and Prioritization , 2007, 2007 IEEE International Conference on Software Maintenance.

[47]  Brian Nielsen,et al.  Formal Approaches to Software Testing, 4th International Workshop, FATES 2004, Linz, Austria, September 21, 2004, Revised Selected Papers , 2005, FATES.

[48]  Marcelo d'Amorim,et al.  Symbolic Execution with Interval Solving and Meta-heuristic Search , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[49]  Nikolaj Bjørner,et al.  Path Feasibility Analysis for String-Manipulating Programs , 2009, TACAS.

[50]  Charles J. Colbourn,et al.  Locating and detecting arrays for interaction faults , 2008, J. Comb. Optim..

[51]  James Miller,et al.  Centroidal Voronoi Tessellations- A New Approach to Random Testing , 2013, IEEE Transactions on Software Engineering.

[52]  Peter Gorm Larsen,et al.  An overview of the ISO/VDM-SL standard , 1992, SIGP.

[53]  Zhenkai Liang,et al.  Darwin: an approach for debugging evolving programs , 2009, ESEC/FSE '09.

[54]  Adam Kiezun,et al.  jFuzz: A Concolic Whitebox Fuzzer for Java , 2009, NASA Formal Methods.

[55]  Mark Harman,et al.  FloPSy - Search-Based Floating Point Constraint Solving for Symbolic Execution , 2010, ICTSS.

[56]  Eitan Farchi,et al.  Using binary decision diagrams for combinatorial test design , 2011, ISSTA '11.

[57]  Corina S. Pasareanu,et al.  Symbolic PathFinder: symbolic execution of Java bytecode , 2010, ASE.

[58]  Leon Sterling,et al.  The Art of Prolog - Advanced Programming Techniques, 2nd Ed , 1986 .

[59]  Mauro Pezzè,et al.  Software testing and analysis - process, principles and techniques , 2007 .

[60]  Myra B. Cohen,et al.  A deterministic density algorithm for pairwise interaction coverage , 2004, IASTED Conf. on Software Engineering.

[61]  Tadashi Dohi,et al.  Enhancing Performance of Random Testing through Markov Chain Monte Carlo Methods , 2010, IEEE Transactions on Computers.

[62]  Thierry Jéron,et al.  TGV : theory , principles and algorithms A tool for the automatic synthesis of conformance test cases for non-deterministic reactive systems , 2004 .

[63]  Phil McMinn,et al.  Search-Based Test Input Generation for String Data Types Using the Results of Web Queries , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[64]  Tao Xie,et al.  Evacon: a framework for integrating evolutionary and concolic testing for object-oriented programs , 2007, ASE.

[65]  Mark Harman,et al.  A multi-objective approach to search-based test data generation , 2007, GECCO '07.

[66]  Jan Tretmans,et al.  04371 Summary - Perspectives of Model-Based Testing , 2004, Perspectives of Model-Based Testing.

[67]  Willem Visser,et al.  Variably interprocedural program analysis for runtime error detection , 2007, ISSTA '07.

[68]  Wolfgang Grieskamp,et al.  A schema language for coordinating construction and composition of partial behavior descriptions , 2006, SCESM '06.

[69]  Frits W. Vaandrager,et al.  Learning I/O Automata , 2010, CONCUR.

[70]  Fabrice Bouquet,et al.  A subset of precise UML for model-based testing , 2007, A-MOST '07.

[71]  Dave Towey,et al.  Restricted Random Testing: Adaptive Random Testing by Exclusion , 2006, Int. J. Softw. Eng. Knowl. Eng..

[72]  Michael Rovatsos,et al.  AAMAS '09 Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems - Volume 2 , 2009, AAMAS 2009.

[73]  Lionel C. Briand,et al.  Using genetic algorithms and coupling measures to devise optimal integration test orders , 2002, SEKE '02.

[74]  Fei-Ching Kuo On adaptive random testing , 2006 .

[75]  David Brumley,et al.  Automatic Patch-Based Exploit Generation is Possible: Techniques and Implications , 2008, 2008 IEEE Symposium on Security and Privacy (sp 2008).

[76]  James Miller,et al.  A Novel Evolutionary Approach for Adaptive Random Testing , 2009, IEEE Transactions on Reliability.

[77]  Thomas Santen,et al.  Automating Test Case Generation from Z Specifications with Isabelle , 1997, ZUM.

[78]  Myra B. Cohen,et al.  Incremental covering array failure characterization in large configuration spaces , 2009, ISSTA.

[79]  Tsun S. Chow,et al.  Testing Software Design Modeled by Finite-State Machines , 1978, IEEE Transactions on Software Engineering.

[80]  Yu Lei,et al.  A Test Generation Strategy for Pairwise Testing , 2002, IEEE Trans. Software Eng..

[81]  D. Richard Kuhn,et al.  Pseudo-Exhaustive Testing for Software , 2006, 2006 30th Annual IEEE/NASA Software Engineering Workshop.

[82]  Angelo Gargantini,et al.  Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing , 2009, TAP@TOOLS.

[83]  James C. King A new approach to program testing , 1975 .

[84]  Thomas A. Henzinger,et al.  Interface automata , 2001, ESEC/FSE-9.

[85]  Bertrand Meyer,et al.  ARTOO: adaptive random testing for object-oriented software , 2008, ICSE.

[86]  A. Jefferson Offutt,et al.  Generating Tests from UML Specifications , 1999, UML.

[87]  Margus Veanes,et al.  Composition of Model Programs , 2007, FORTE.

[88]  Mark Harman,et al.  Open Problems in Testability Transformation , 2008, 2008 IEEE International Conference on Software Testing Verification and Validation Workshop.

[89]  Gregg Rothermel ISSTA 2004 : proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis, Boston, Massachusetts, USA, July 11-14, 2004 , 2004 .

[90]  Rupak Majumdar,et al.  Hybrid Concolic Testing , 2007, 29th International Conference on Software Engineering (ICSE'07).

[91]  Jan Tretmans,et al.  Test Generation with Inputs, Outputs and Repetitive Quiescence , 1996, Softw. Concepts Tools.

[92]  Tsong Yueh Chen,et al.  On the statistical properties of testing effectiveness measures , 2006, J. Syst. Softw..

[93]  Wolfgang Grieskamp,et al.  Model‐based quality assurance of protocol documentation: tools and methodology , 2011, Softw. Test. Verification Reliab..

[94]  Matthias Grochtmann,et al.  Verifying Timing Constraints of Real-Time Systems by Means of Evolutionary Testing , 1998, Real-Time Systems.

[95]  Alexander Pretschner,et al.  10421 Summary - Model-Based Testing in Practice , 2010, Model-Based Testing in Practice.

[96]  Corina S. Pasareanu,et al.  A survey of new trends in symbolic execution for software testing and analysis , 2009, International Journal on Software Tools for Technology Transfer.

[97]  Leon Sterling,et al.  The Art of Prolog - Advanced Programming Techniques , 1986 .

[98]  Robert Brownlie,et al.  Robust testing of AT&T PMX/StarMAIL using OATS , 1992, AT&T Technical Journal.

[99]  Minh Ngoc Ngo,et al.  Detecting large number of infeasible paths through recognizing their patterns , 2007, ESEC-FSE '07.

[100]  Miguel Castro,et al.  Better bug reporting with better privacy , 2008, ASPLOS 2008.

[101]  Alan Hartman,et al.  Projected state machine coverage for software testing , 2002, ISSTA '02.

[102]  Myra B. Cohen,et al.  Configuration-aware regression testing: an empirical study of sampling and prioritization , 2008, ISSTA '08.

[103]  R. A. Fisher,et al.  Design of Experiments , 1936 .

[104]  Andreas Zeller,et al.  Exploiting Common Object Usage in Test Case Generation , 2011, 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation.

[105]  Myra B. Cohen,et al.  Evaluating improvements to a meta-heuristic search for constrained interaction testing , 2011, Empirical Software Engineering.

[106]  N. H. Beebe on Software Tools for Technology Transfer ( STTT ) , 2005 .

[107]  Lionel C. Briand,et al.  A Systematic Review of the Application and Empirical Investigation of Search-Based Test Case Generation , 2010, IEEE Transactions on Software Engineering.

[108]  Bertrand Jeannet,et al.  Symbolic Test Selection Based on Approximate Analysis , 2005, TACAS.

[109]  Koen Claessen,et al.  QuickCheck: a lightweight tool for random testing of Haskell programs , 2011, SIGP.

[110]  Lawrence C. Paulson,et al.  Isabelle - A Generic Theorem Prover (with a contribution by T. Nipkow) , 1994, Lecture Notes in Computer Science.

[111]  Sebastian Wieczorek,et al.  Improving Testing of Enterprise Systems by Model-Based Testing on Graphical User Interfaces , 2010, 2010 17th IEEE International Conference and Workshops on Engineering of Computer Based Systems.

[112]  Margus Veanes,et al.  Rex: Symbolic Regular Expression Explorer , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[113]  A. Arcuri Automated System Testing of Real-Time Embedded Systems Based on Environment Models 1 , 2011 .

[114]  Stephen McCamant,et al.  Loop-extended symbolic execution on binary programs , 2009, ISSTA.

[115]  Kari J. Nurmela,et al.  Upper bounds for covering arrays by tabu search , 2004, Discret. Appl. Math..

[116]  Bruno Dutertre,et al.  A Fast Linear-Arithmetic Solver for DPLL(T) , 2006, CAV.

[117]  Mary Jean Harrold,et al.  Heap cloning: Enabling dynamic symbolic execution of java programs , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[118]  John A. Clark,et al.  Search-based mutation testing for Simulink models , 2005, GECCO '05.

[119]  Mark Harman,et al.  Input Domain Reduction through Irrelevant Variable Removal and Its Effect on Local, Global, and Hybrid Search-Based Structural Test Data Generation , 2012, IEEE Transactions on Software Engineering.

[120]  Michel Gendreau,et al.  Handbook of Metaheuristics , 2010 .

[121]  Bruno Legeard,et al.  Automated Boundary Testing from Z and B , 2002, FME.

[122]  Mario Cannataro,et al.  Protein-to-protein interactions: Technologies, databases, and algorithms , 2010, CSUR.

[123]  Jinhui Shan,et al.  An Approach to Test Data Generation for Killing Multiple Mutants , 2006, 2006 22nd IEEE International Conference on Software Maintenance.

[124]  Margus Veanes,et al.  Generating finite state machines from abstract state machines , 2002, ISSTA '02.

[125]  Alexander K. Petrenko,et al.  The UniTesK Approach to Designing Test Suites , 2003, Programming and Computer Software.

[126]  Huai Liu,et al.  Adaptive random testing based on distribution metrics , 2009, J. Syst. Softw..

[127]  Jacques Klein,et al.  Automated and Scalable T-wise Test Case Generation Strategies for Software Product Lines , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[128]  Paul Clements,et al.  Software product lines - practices and patterns , 2001, SEI series in software engineering.

[129]  Robert M. Hierons,et al.  The Effect of the Distributed Test Architecture on the Power of Testing , 2008, Comput. J..

[130]  Robert Mandl,et al.  Orthogonal Latin squares: an application of experiment design to compiler testing , 1985, CACM.

[131]  Nikolai Tillmann,et al.  Action Machines: a Framework for Encoding and Composing Partial Behaviors , 2006, Int. J. Softw. Eng. Knowl. Eng..

[132]  Gary McGraw,et al.  Generating Software Test Data by Evolution , 2001, IEEE Trans. Software Eng..

[133]  Michael L. Fredman,et al.  The AETG System: An Approach to Testing Based on Combinatiorial Design , 1997, IEEE Trans. Software Eng..

[134]  D.M. Cohen,et al.  The Combinatorial Design Approach to Automatic Test Generation , 1996, IEEE Softw..

[135]  Robert G. Merkel,et al.  Analysis and enhancements of adaptive random testing , 2005 .

[136]  Margus Veanes,et al.  Instrumenting scenarios in a model-driven development environment , 2004, Inf. Softw. Technol..

[137]  Alfred V. Aho,et al.  An optimization technique for protocol conformance test generation based on UIO sequences and rural Chinese postman tours , 1991, IEEE Trans. Commun..

[138]  Nikolai Tillmann,et al.  Pex-White Box Test Generation for .NET , 2008, TAP.

[139]  Sarfraz Khurshid,et al.  Generalized Symbolic Execution for Model Checking and Testing , 2003, TACAS.

[140]  Charles J. Colbourn,et al.  Prioritized interaction testing for pair-wise coverage with seeding and constraints , 2006, Inf. Softw. Technol..

[141]  K. Larsen,et al.  Online Testing of Real-time Systems Using Uppaal , 2004, FATES.

[142]  Mark Harman,et al.  Automated Unique Input Output Sequence Generation for Conformance Testing of FSMs , 2006, Comput. J..

[143]  Thomas A. Henzinger,et al.  Alternating Refinement Relations , 1998, CONCUR.

[144]  Jan Tretmans,et al.  TorX: Automated Model-Based Testing , 2003 .

[145]  Nikolai Tillmann,et al.  XRT- Exploring Runtime for .NET Architecture and Applications , 2006, Electron. Notes Theor. Comput. Sci..

[146]  Nicolae Goga,et al.  Test Selection, Trace Distance and Heuristics , 2002, TestCom.

[147]  Paul D. Seymour,et al.  AETG/sup SM/ Web: a Web based service for automatic efficient test generation from functional requirements , 1998, Proceedings. 2nd IEEE Workshop on Industrial Strength Formal Specification Techniques.

[148]  Robert M. Hierons,et al.  Reaching and Distinguishing States of Distributed Systems , 2010, SIAM J. Comput..

[149]  Laurent Fribourg,et al.  Test sets generation from algebraic specifications using logic programming , 1986, J. Syst. Softw..

[150]  R. L. Erickson,et al.  Improved quality of protocol testing through techniques of experimental design , 1994, Proceedings of ICC/SUPERCOMM'94 - 1994 International Conference on Communications.

[151]  Johannes Mayer,et al.  Lattice-based adaptive random testing , 2005, ASE.

[152]  Margus Veanes,et al.  Testing Concurrent Object-Oriented Systems with Spec Explorer , 2005, FM.

[153]  Rupak Majumdar,et al.  Reducing Test Inputs Using Information Partitions , 2009, CAV.

[154]  Tsong Yueh Chen,et al.  Adaptive random testing through dynamic partitioning , 2004 .

[155]  Myra B. Cohen,et al.  Constructing test suites for interaction testing , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[156]  William G. Cochran,et al.  Experimental Designs, 2nd Edition , 1950 .

[157]  Andreas Zeller,et al.  Mutation-Driven Generation of Unit Tests and Oracles , 2012, IEEE Trans. Software Eng..

[158]  Robert M. Hierons,et al.  Conformance Testing from Message Sequence Charts , 2011, 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation.

[159]  Phil McMinn,et al.  Search‐based software test data generation: a survey , 2004, Softw. Test. Verification Reliab..

[160]  Mark Harman,et al.  Automated test data generation for aspect-oriented programs , 2009, AOSD '09.

[161]  T. Y. Chen,et al.  Adaptive Random Testing , 2004, ASIAN.

[162]  Mark Harman,et al.  Handling dynamic data structures in search based testing , 2008, GECCO '08.

[163]  Angelo Gargantini,et al.  A Formal Logic Approach to Constrained Combinatorial Testing , 2010, Journal of Automated Reasoning.

[164]  Yu-Wen Tung,et al.  Automating test case generation for the new generation mission software system , 2000, 2000 IEEE Aerospace Conference. Proceedings (Cat. No.00TH8484).

[165]  Myra B. Cohen,et al.  Coverage and adequacy in software product line testing , 2006, ROSATEA '06.

[166]  George Candea,et al.  S2E: a platform for in-vivo multi-path analysis of software systems , 2011, ASPLOS XVI.

[167]  Sarfraz Khurshid,et al.  Generalizing symbolic execution to library classes , 2005, PASTE '05.

[168]  Myra B. Cohen,et al.  Feedback driven adaptive combinatorial testing , 2011, ISSTA '11.

[169]  Myra B. Cohen,et al.  Augmenting simulated annealing to build interaction test suites , 2003, 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003..

[170]  Alan Hartman,et al.  Problems and algorithms for covering arrays , 2004, Discret. Math..

[171]  Koushik Sen DART: Directed Automated Random Testing , 2009, Haifa Verification Conference.

[172]  Jean-Raymond Abrial,et al.  The B-book - assigning programs to meanings , 1996 .

[173]  Myra B. Cohen,et al.  Covering arrays for efficient fault characterization in complex configuration spaces , 2004, IEEE Transactions on Software Engineering.

[174]  Mark Harman,et al.  n empirical investigation into branch coverage for C programs using CUTE and USTIN , 2010 .

[175]  Aurora Trinidad Ramirez Pozo,et al.  Integration Test of Classes and Aspects with a Multi-Evolutionary and Coupling-Based Approach , 2011, SSBSE.

[176]  Lionel C. Briand,et al.  Reducing the Cost of Model-Based Testing through Test Case Diversity , 2010, ICTSS.

[177]  Andrea Arcuri,et al.  On the automation of fixing software bugs , 2008, ICSE Companion '08.

[178]  Jan Tretmans,et al.  Model Based Testing with Labelled Transition Systems , 2008, Formal Methods and Testing.

[179]  C. L. Mallows,et al.  Applying Design of Experiments to Software Testing , 1997, Proceedings of the (19th) International Conference on Software Engineering.

[180]  Nikolai Tillmann,et al.  Demand-Driven Compositional Symbolic Execution , 2008, TACAS.

[181]  Xin Yao,et al.  A novel co-evolutionary approach to automatic software bug fixing , 2008, 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence).

[182]  Wasif Afzal,et al.  A systematic review of search-based testing for non-functional system properties , 2009, Inf. Softw. Technol..

[183]  Mark Harman,et al.  Empirical evaluation of a nesting testability transformation for evolutionary testing , 2009, TSEM.

[184]  James Miller,et al.  Automated cookie collection testing , 2014, TSEM.

[185]  Jianjun Zhao,et al.  A Divergence-Oriented Approach to Adaptive Random Testing of Java Programs , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[186]  Dawson R. Engler,et al.  RWset: Attacking Path Explosion in Constraint-Based Test Generation , 2008, TACAS.

[187]  Tsong Yueh Chen,et al.  Proportional sampling strategy: a compendium and some insights , 2001, J. Syst. Softw..

[188]  Lionel C. Briand,et al.  Adaptive random testing: an illusion of effectiveness? , 2011, ISSTA '11.

[189]  Sarfraz Khurshid,et al.  Symbolic execution for software testing in practice: preliminary assessment , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[190]  Mark Harman,et al.  How to Overcome the Equivalent Mutant Problem and Achieve Tailored Selective Mutation Using Co-evolution , 2004, GECCO.

[191]  Enrique Alba,et al.  Management of Software Projects with GAs , 2005 .

[192]  Mark Harman,et al.  Optimizing for the Number of Tests Generated in Search Based Test Data Generation with an Application to the Oracle Cost Problem , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[193]  Raúl A. Santelices,et al.  Exploiting program dependencies for scalable multiple-path symbolic execution , 2010, ISSTA '10.

[194]  Hartmut Ehrig,et al.  Fundamentals of Algebraic Specification 1: Equations and Initial Semantics , 1985 .

[195]  William G. Cochran,et al.  Experimental designs, 2nd ed. , 1957 .

[196]  Corina S. Pasareanu,et al.  Symbolic execution with mixed concrete-symbolic solving , 2011, ISSTA '11.

[197]  Jüri Vain,et al.  Generating Tests from EFSM Models Using Guided Model Checking and Iterated Search Refinement , 2006, FATES/RV.

[198]  Bo Zhou,et al.  Enhancing Performance of Random Testing through Markov Chain Monte Carlo Methods , 2013, IEEE Trans. Computers.

[199]  Burkhart Wolff,et al.  Formal Approaches to Software Testing and Runtime Verification: First Combined International Workshops FATES 2006 and RV 2006 Seattle, WA, USA, ... Papers (Lecture Notes in Computer Science) , 2007 .

[200]  Jacek Czerwonka,et al.  Pairwise Testing in Real World , 2006 .

[201]  Alexandre Petrenko,et al.  Covering transitions of concurrent systems through queues , 2005, 16th IEEE International Symposium on Software Reliability Engineering (ISSRE'05).

[202]  Mark utting Model-Based Testing – Next Generation Functional Software Testing , 2010 .

[203]  Lionel C. Briand,et al.  A UML-Based Approach to System Testing , 2001, UML.

[204]  James C. King,et al.  A new approach to program testing , 1974, Programming Methodology.

[205]  Margus Veanes,et al.  Test Case Generation from AsmL Specifications , 2003, Abstract State Machines.

[206]  Patrice Godefroid,et al.  Compositional Dynamic Test Generation (Extended Abstract) , 2007 .

[207]  Joachim Wegener,et al.  Evaluation of Different Fitness Functions for the Evolutionary Testing of an Autonomous Parking System , 2004, GECCO.

[208]  Koushik Sen,et al.  CUTE and jCUTE: Concolic Unit Testing and Explicit Path Model-Checking Tools , 2006, CAV.

[209]  David L. Dill,et al.  A Decision Procedure for Bit-Vectors and Arrays , 2007, CAV.

[210]  Michael R. Lowry,et al.  Combining unit-level symbolic execution and system-level concrete execution for testing nasa software , 2008, ISSTA '08.

[211]  A. Baron Experimental Designs , 1990, The Behavior analyst.

[212]  Achim D. Brucker,et al.  On theorem prover-based testing , 2012, Formal Aspects of Computing.

[213]  Corina S. Pasareanu,et al.  Symbolic execution with abstraction , 2008, International Journal on Software Tools for Technology Transfer.

[214]  Olivier Charles,et al.  Relating Conformance Test Coverage to Formal Specifications , 1996, FORTE.

[215]  Marc J. Balcer,et al.  The category-partition method for specifying and generating fuctional tests , 1988, CACM.

[216]  Yuanyuan Zhang,et al.  Search Based Software Engineering: A Comprehensive Analysis and Review of Trends Techniques and Applications , 2009 .

[217]  Matthew B. Dwyer,et al.  Automatic generation of load tests , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[218]  Lionel C. Briand,et al.  Empirical Investigation of the Effects of Test Suite Properties on Similarity-Based Test Case Selection , 2011, 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation.

[219]  Myra B. Cohen,et al.  Constructing Interaction Test Suites for Highly-Configurable Systems in the Presence of Constraints: A Greedy Approach , 2008, IEEE Transactions on Software Engineering.

[220]  Tsong Yueh Chen,et al.  Adaptive Random Testing: The ART of test case diversity , 2010, J. Syst. Softw..

[221]  Myra B. Cohen,et al.  GUI Interaction Testing: Incorporating Event Context , 2011, IEEE Transactions on Software Engineering.

[222]  Mark Harman,et al.  Strong higher order mutation-based test data generation , 2011, ESEC/FSE '11.

[223]  Myra B. Cohen,et al.  A variable strength interaction testing of components , 2003, Proceedings 27th Annual International Computer Software and Applications Conference. COMPAC 2003.

[224]  Marcelo d'Amorim,et al.  CORAL: Solving Complex Constraints for Symbolic PathFinder , 2011, NASA Formal Methods.

[225]  Hong Zhu,et al.  Software unit test coverage and adequacy , 1997, ACM Comput. Surv..

[226]  Adam Kiezun,et al.  Grammar-based whitebox fuzzing , 2008, PLDI '08.

[227]  Hareton K. N. Leung,et al.  A survey of combinatorial testing , 2011, CSUR.

[228]  Marie-Claude Gaudel,et al.  Testing Can Be Formal, Too , 1995, TAPSOFT.

[229]  Antonia Bertolino,et al.  Software Testing Research: Achievements, Challenges, Dreams , 2007, Future of Software Engineering (FOSE '07).

[230]  John D. McGregor,et al.  Testing a Software Product Line , 2001, PSSE.

[231]  Paolo Tonella,et al.  Symbolic search-based testing , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[232]  Margus Veanes,et al.  Model-Based Testing of Object-Oriented Reactive Systems with Spec Explorer , 2008, Formal Methods and Testing.

[233]  Yashwant K. Malaiya,et al.  Antirandom testing: getting the most out of black-box testing , 1995, Proceedings of Sixth International Symposium on Software Reliability Engineering. ISSRE'95.

[234]  D. Richard Kuhn,et al.  Software fault interactions and implications for software testing , 2004, IEEE Transactions on Software Engineering.

[235]  David Lee,et al.  Principles and methods of testing finite state machines-a survey , 1996, Proc. IEEE.

[236]  Alessandro Orso,et al.  Type-Dependence Analysis and Program Transformation for Symbolic Execution , 2007, TACAS.

[237]  Tsong Yueh Chen,et al.  On Favourable Conditions for Adaptive Random Testing , 2007, Int. J. Softw. Eng. Knowl. Eng..

[238]  Mark Harman,et al.  A Theoretical and Empirical Study of Search-Based Testing: Local, Global, and Hybrid Search , 2010, IEEE Transactions on Software Engineering.

[239]  Alessandro Orso,et al.  Test-Suite Augmentation for Evolving Software , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[240]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.

[241]  Chen Fu,et al.  Is Data Privacy Always Good for Software Testing? , 2010, 2010 IEEE 21st International Symposium on Software Reliability Engineering.

[242]  Patrice Godefroid,et al.  Automatic partial loop summarization in dynamic test generation , 2011, ISSTA '11.

[243]  Nikolaj Bjørner,et al.  Alternating Simulation and IOCO , 2010, ICTSS.

[244]  Jan Tretmans,et al.  Test Generation Based on Symbolic Specifications , 2004, FATES.

[245]  C. L. Mallows,et al.  Applying design of experiments to software testing: experience report , 1997, ICSE '97.

[246]  Mika Katara,et al.  Making Model-Based Testing More Agile: A Use Case Driven Approach , 2006, Haifa Verification Conference.

[247]  Rupak Majumdar,et al.  Symbolic Robustness Analysis , 2009, 2009 30th IEEE Real-Time Systems Symposium.