Automated Software Test Generation: Some Challenges, Solutions, and Recent Advances

The automation of software testing promises to delegate to machines what is otherwise the most labor-intensive and expensive part of software development. The past decade has seen a resurgence in research interest for this problem, bringing about significant progress. In this article, we provide an overview of automated test generation for software, and then discuss recent developments that have had significant impact on real-life software.

[1]  George Candea,et al.  The S2E Platform: Design, Implementation, and Applications , 2012, TOCS.

[2]  D. Wagner,et al.  Catchconv : Symbolic execution and run-time type inference for integer conversion errors , 2007 .

[3]  Zijiang Yang,et al.  F-Soft: Software Verification Platform , 2005, CAV.

[4]  William R. Bush,et al.  A static analyzer for finding dynamic programming errors , 2000, Softw. Pract. Exp..

[5]  Emin Gün Sirer,et al.  Using production grammars in software testing , 1999, DSL '99.

[6]  Doron Drusinsky,et al.  The Temporal Rover and the ATG Rover , 2000, SPIN.

[7]  Shuvendu K. Lahiri,et al.  From Program to Logic: An Introduction , 2011, LASER Summer School.

[8]  David Brumley,et al.  AEG: Automatic Exploit Generation , 2011, NDSS.

[9]  Daniel Kroening,et al.  A Tool for Checking ANSI-C Programs , 2004, TACAS.

[10]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.

[11]  Ralf Lämmel,et al.  Controllable Combinatorial Coverage in Grammar-Based Testing , 2006, TestCom.

[12]  Adam Kiezun,et al.  Grammar-based whitebox fuzzing , 2008, PLDI '08.

[13]  Patrice Godefroid,et al.  Compositional dynamic test generation , 2007, POPL '07.

[14]  Darko Marinov,et al.  Automated testing of refactoring engines , 2007, ESEC-FSE '07.

[15]  Grigore Rosu,et al.  Monitoring Java Programs with Java PathExplorer , 2001, RV@CAV.

[16]  Dawson R. Engler,et al.  Execution Generated Test Cases: How to Make Systems Code Crash Itself , 2005, SPIN.

[17]  George Candea,et al.  Testing Closed-Source Binary Device Drivers with DDT , 2010, USENIX Annual Technical Conference.

[18]  Phil McMinn,et al.  Search‐based software test data generation: a survey , 2004, Softw. Test. Verification Reliab..

[19]  Xavier Rival,et al.  Trace Partitioning in Abstract Interpretation Based Static Analyzers , 2005, ESOP.

[20]  Peter M. Maurer,et al.  Generating test data with enhanced context-free grammars , 1990, IEEE Software.

[21]  Derek Bruening,et al.  AddressSanitizer: A Fast Address Sanity Checker , 2012, USENIX Annual Technical Conference.

[22]  Barton P. Miller,et al.  An empirical study of the reliability of UNIX utilities , 1990, Commun. ACM.

[23]  Peter Schachte,et al.  State Joining and Splitting for the Symbolic Execution of Binaries , 2009, RV.

[24]  Koushik Sen.  DART: Directed Automated Random Testing , 2009, Haifa Verification Conference.

[25]  C. V. Ramamoorthy,et al.  On the Automated Generation of Program Test Data , 1976, IEEE Transactions on Software Engineering.

[26]  Patrice Godefroid,et al.  Automated Whitebox Fuzz Testing , 2008, NDSS.

[27]  Bogdan Korel,et al.  A dynamic approach of test data generation , 1990, Proceedings. Conference on Software Maintenance 1990.

[28]  George Candea,et al.  Parallel symbolic execution for automated real-world software testing , 2011, EuroSys '11.

[29]  Asim Kadav,et al.  SymDrive: Testing Drivers without Devices , 2012, OSDI.

[30]  Dawson R. Engler,et al.  A system and language for building system-specific, static analyses , 2002, PLDI '02.

[31]  Barton P. Miller,et al.  An empirical study of the robustness of Windows NT applications using random testing , 2000 .

[32]  Rupak Majumdar,et al.  Directed test generation using symbolic grammars , 2007, ASE.

[33]  Edsger W. Dijkstra,et al.  Guarded commands, nondeterminacy and formal derivation of programs , 1975, Commun. ACM.

[34]  David Coppit,et al.  yagg: an easy-to-use generator for structured test inputs , 2005, ASE.

[35]  Nikolai Tillmann,et al.  Demand-Driven Compositional Symbolic Execution , 2008, TACAS.

[36]  Sriram K. Rajamani,et al.  Compositional may-must program analysis: unleashing the power of alternation , 2010, POPL '10.

[37]  Bruno Marre,et al.  PathCrawler: Automatic Generation of Path Tests by Combining Static and Dynamic Analysis , 2005, EDCC.

[38]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[39]  Dawson R. Engler,et al.  EXE: automatically generating inputs of death , 2006, CCS '06.

[40]  Nikolai Tillmann,et al.  Pex-White Box Test Generation for .NET , 2008, TAP.

[41]  Cacm Staff,et al.  BufferBloat , 2011, Communications of the ACM.

[42]  Mark R. Tuttle,et al.  Symbolic Execution for BIOS Security , 2015, WOOT.

[43]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[44]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[45]  Bruno Dutertre,et al.  A Fast Linear-Arithmetic Solver for DPLL(T) , 2006, CAV.

[46]  Frank Tip,et al.  Finding Bugs in Web Applications Using Dynamic Test Generation and Explicit-State Model Checking , 2010, IEEE Transactions on Software Engineering.

[47]  Bor-Yuh Evan Chang,et al.  Boogie: A Modular Reusable Verifier for Object-Oriented Programs , 2005, FMCO.

[48]  George Candea,et al.  Automated software testing as a service , 2010, SoCC '10.

[49]  Lori A. Clarke,et al.  A program testing system , 1976, ACM '76.

[50]  Aarti Gupta,et al.  Tunneling and slicing: Towards scalable BMC , 2008, 2008 45th ACM/IEEE Design Automation Conference.

[51]  Fabrice Bellard,et al.  QEMU, a Fast and Portable Dynamic Translator , 2005, USENIX Annual Technical Conference, FREENIX Track.

[52]  Katerina J. Argyraki,et al.  Software dataplane verification , 2014, NSDI.

[53]  Thomas W. Reps,et al.  Precise interprocedural dataflow analysis via graph reachability , 1995, POPL '95.

[54]  Thomas Ball,et al.  Finding and Reproducing Heisenbugs in Concurrent Programs , 2008, OSDI.

[55]  Bengt Jonsson,et al.  Combining Black-Box and White-Box Techniques for Learning Register Automata , 2019, Computing and Software Science.

[56]  Patrice Godefroid,et al.  Fuzzing in The Cloud (Position Statement) , 2010 .

[57]  Koen Claessen,et al.  QuickCheck: a lightweight tool for random testing of Haskell programs , 2011, SIGP.

[58]  Rupak Majumdar,et al.  Reducing Test Inputs Using Information Partitions , 2009, CAV.

[59]  Hardi Hungar,et al.  Model Generation by Moderated Regular Extrapolation , 2002, FASE.

[60]  Edmund M. Clarke,et al.  Symbolic Model Checking: 10^20 States and Beyond , 1990, Inf. Comput..

[61]  Michael Howard,et al.  The security development lifecycle : SDL, a process for developing demonstrably more secure software , 2006 .

[62]  George Candea,et al.  -OVERIFY: Optimizing Programs for Fast Verification , 2013, HotOS.

[63]  Madan Musuvathi,et al.  Iterative context bounding for systematic testing of multithreaded programs , 2007, PLDI '07.

[64]  William E. Howden,et al.  Symbolic Testing and the DISSECT Symbolic Evaluation System , 1977, IEEE Transactions on Software Engineering.

[65]  George Candea,et al.  Reverse engineering of binary device drivers with RevNIC , 2010, EuroSys '10.

[66]  Pedram Amini,et al.  Fuzzing: Brute Force Vulnerability Discovery , 2007 .

[67]  Konstantin Serebryany,et al.  ThreadSanitizer: data race detection in practice , 2009, WBIA '09.

[68]  P. Purdom.  A sentence generator for testing parsers , 1972 .

[69]  Karl N. Levitt,et al.  SELECT—a formal system for testing and debugging programs by symbolic execution , 1975 .

[70]  Silvio Ghilardi,et al.  Decision Procedures for Flat Array Properties , 2014, Journal of Automated Reasoning.

[71]  Alan J. Hu,et al.  Calysto: scalable and precise extended static checking , 2008, ICSE.

[72]  Armin Biere,et al.  Bounded Model Checking Using Satisfiability Solving , 2001, Formal Methods Syst. Des..

[73]  Mary Lou Soffa,et al.  Generating test data for branch coverage , 2000, Proceedings ASE 2000. Fifteenth IEEE International Conference on Automated Software Engineering.

[74]  D.M. Cohen,et al.  The Combinatorial Design Approach to Automatic Test Generation , 1996, IEEE Softw..

[75]  Steve Hanna,et al.  A Symbolic Execution Framework for JavaScript , 2010, 2010 IEEE Symposium on Security and Privacy.

[76]  Sorin Lerner,et al.  ESP: path-sensitive program verification in polynomial time , 2002, PLDI '02.

[77]  Koushik Sen,et al.  Heuristics for Scalable Dynamic Test Generation , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[78]  K. V. Hanford,et al.  Automatic Generation of Test Cases , 1970, IBM Syst. J..

[79]  Eitan Farchi,et al.  Framework for testing multi‐threaded Java programs , 2003, Concurr. Comput. Pract. Exp..

[80]  A. Jefferson Offutt,et al.  The dynamic domain reduction procedure for test data generation , 1999, Softw. Pract. Exp..

[81]  Todd C. Mowry.  Introduction to Special Issue APLOS 2011 , 2012, TOCS.

[82]  Tiziana Margaria,et al.  Dynamic testing via automata learning , 2007, International Journal on Software Tools for Technology Transfer.

[83]  David Brumley,et al.  Automatic Patch-Based Exploit Generation is Possible: Techniques and Implications , 2008, 2008 IEEE Symposium on Security and Privacy (sp 2008).

[84]  Patrice Godefroid,et al.  Dynamic partial-order reduction for model checking software , 2005, POPL '05.

[85]  Armin Biere,et al.  Boolector: An Efficient SMT Solver for Bit-Vectors and Arrays , 2009, TACAS.

[86]  Mark Lillibridge,et al.  Extended static checking for Java , 2002, PLDI '02.

[87]  Patrice Godefroid.  Higher-order test generation , 2011, PLDI '11.

[88]  Alexander Aiken,et al.  Scalable error detection using boolean satisfiability , 2005, POPL '05.

[89]  George Candea,et al.  Efficient state merging in symbolic execution , 2012, Software Engineering.

[90]  George Candea,et al.  S2E: a platform for in-vivo multi-path analysis of software systems , 2011, ASPLOS XVI.

[91]  Zvonimir Rakamaric,et al.  JDart: A Dynamic Symbolic Analysis Framework , 2016, TACAS.

[92]  Lori A. Clarke,et al.  Applications of symbolic evaluation , 1985, J. Syst. Softw..

[93]  Patrice Godefroid,et al.  Billions and billions of constraints: Whitebox fuzz testing in production , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[94]  David Brumley,et al.  Unleashing Mayhem on Binary Code , 2012, 2012 IEEE Symposium on Security and Privacy.

[95]  Bruno Legeard,et al.  A taxonomy of model‐based testing approaches , 2012, Softw. Test. Verification Reliab..

[96]  Edsger W. Dijkstra,et al.  Notes on structured programming , 1970 .

[97]  David A. Wagner,et al.  Model Checking One Million Lines of C Code , 2004, NDSS.

[98]  Zhenkai Liang,et al.  BitBlaze: A New Approach to Computer Security via Binary Analysis , 2008, ICISS.

[99]  George Candea,et al.  Prototyping symbolic execution engines for interpreted languages , 2014, ASPLOS.

[100]  Dinakar Dhurjati,et al.  SAFECode: enforcing alias analysis for weakly typed languages , 2005, PLDI '06.

[101]  Patrice Godefroid,et al.  Model checking for programming languages using VeriSoft , 1997, POPL '97.

[102]  David Mosberger,et al.  Memory consistency models , 1993, OPSR.

[103]  Thomas Ball,et al.  The Static Driver Verifier Research Platform , 2010, CAV.

[104]  David L. Dill,et al.  A Decision Procedure for Bit-Vectors and Arrays , 2007, CAV.