A guided genetic algorithm for automated crash reproduction

To reduce the effort developers have to make for crash debugging, researchers have proposed several solutions for automatic failure reproduction. Recent advances proposed the use of symbolic execution, mutation analysis, and directed model checking as underling techniques for post-failure analysis of crash stack traces. However, existing approaches still cannot reproduce many real-world crashes due to such limitations as environment dependencies, path explosion, and time complexity. To address these challenges, we present EvoCrash, a post-failure approach which uses a novel Guided Genetic Algorithm (GGA) to cope with the large search space characterizing real-world software programs. Our empirical study on three open-source systems shows that EvoCrash can replicate 41 (82%) of real-world crashes, 34 (89%) of which are useful reproductions for debugging purposes, outperforming the state-of-the-art in crash replication.

[1]  John Steven,et al.  jRapture: A Capture/Replay tool for observation-based testing , 2000, ISSTA '00.

[2]  Sarfraz Khurshid,et al.  Korat: automated testing based on Java predicates , 2002, ISSTA '02.

[3]  Phil McMinn,et al.  Search‐based software test data generation: a survey , 2004, Softw. Test. Verification Reliab..

[4]  Satish Narayanasamy,et al.  BugNet: continuously recording program execution for deterministic replay debugging , 2005, 32nd International Symposium on Computer Architecture (ISCA'05).

[5]  Michael D. Ernst,et al.  Automatic test factoring for java , 2005, ASE '05.

[6]  Koushik Sen,et al.  DART: directed automated random testing , 2005, PLDI '05.

[7]  Alessandro Orso,et al.  A Technique for Enabling and Supporting Debugging of Field Failures , 2007, 29th International Conference on Software Engineering (ICSE'07).

[8]  Michael D. Ernst,et al.  Feedback-Directed Random Test Generation , 2007, 29th International Conference on Software Engineering (ICSE'07).

[9]  Bertrand Meyer,et al.  Contract driven development = test driven development - writing test cases , 2007, ESEC-FSE '07.

[10]  Christel Baier,et al.  Principles of model checking , 2008 .

[11]  Nikolai Tillmann,et al.  Pex-White Box Test Generation for .NET , 2008, TAP.

[12]  Christel Baier,et al.  Principles of Model Checking (Representation and Mind Series) , 2008 .

[13]  Michael D. Ernst,et al.  ReCrash: Making Software Failures Reproducible by Preserving Object States , 2008, ECOOP.

[14]  Bertrand Meyer,et al.  On the Effectiveness of Test Extraction without Overhead , 2009, 2009 International Conference on Software Testing Verification and Validation.

[15]  Carl K. Chang,et al.  OCAT: object capture-based automated testing , 2010, ISSTA '10.

[16]  George Candea,et al.  Execution synthesis: a technique for automated software debugging , 2010, EuroSys '10.

[17]  Mark Harman,et al.  Search Based Software Engineering: Techniques, Taxonomy, Tutorial , 2010, LASER Summer School.

[18]  Xiangyu Zhang,et al.  Analyzing multicore dumps to facilitate concurrency bug reproduction , 2010, ASPLOS XV.

[19]  Amiram Yehudai,et al.  Regression Test Selection Techniques for Test-Driven Development , 2011, 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.

[20]  Nikolai Tillmann,et al.  Precise identification of problems for structural test generation , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[21]  Alessandro Orso,et al.  BugRedux: Reproducing field failures for in-house debugging , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[22]  Yuanyuan Zhang,et al.  Search-based software engineering: Trends, techniques and applications , 2012, CSUR.

[23]  Alessandro Orso,et al.  SBFR: A search based approach for reproducing failures of programs with grammar based input , 2013, 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[24]  Gordon Fraser,et al.  Whole Test Suite Generation , 2013, IEEE Transactions on Software Engineering.

[25]  Sheeva Afshan,et al.  Evolving Readable String Test Inputs Using a Natural Language Model to Reduce Human Oracle Cost , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[26]  Gordon Fraser,et al.  EvoSuite: On the Challenges of Test Case Generation in the Real World , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[27]  I. S. W. B. Prasetya T3, a Combinator-Based Random Testing Tool for Java: Benchmarking , 2013, FITTEST@ICTSS.

[28]  Andreas Zeller,et al.  Reconstructing Core Dumps , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[29]  Gordon Fraser,et al.  1600 faults in 100 projects: automatically finding faults while achieving high coverage with EvoSuite , 2015, Empirical Software Engineering.

[30]  Gordon Fraser,et al.  Parameter tuning or default values? An empirical investigation in search-based software engineering , 2013, Empirical Software Engineering.

[31]  Alessandro Orso,et al.  Reproducing Field Failures for Programs with Complex Grammar-Based Input , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[32]  Ning Chen,et al.  STAR: Stack Trace Based Automatic Crash Reproduction via Symbolic Execution , 2015, IEEE Transactions on Software Engineering.

[33]  Paolo Tonella,et al.  Reformulating Branch Coverage as a Many-Objective Optimization Problem , 2015, 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST).

[34]  Abdelwahab Hamou-Lhadj,et al.  JCHARMING: A bug reproduction approach using crash traces and directed model checking , 2015, 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER).

[35]  Andrea De Lucia,et al.  Improving Multi-Objective Test Case Selection by Injecting Diversity in Genetic Algorithms , 2015, IEEE Transactions on Software Engineering.

[36]  Yann-Gaël Guéhéneuc,et al.  Instance Generator and Problem Representation to Improve Object Oriented Code Coverage , 2015, IEEE Transactions on Software Engineering.

[37]  Martin Monperrus,et al.  Crash reproduction via test case mutation: let existing test cases help , 2015, ESEC/SIGSOFT FSE.

[38]  Arie van Deursen,et al.  Evolutionary testing for crash reproduction , 2016, SBST@ICSE.

[39]  Christopher Vendome,et al.  Automatically Discovering, Reporting and Reproducing Android Application Crashes , 2016, 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST).

[40]  Christoph Treude,et al.  Exception handling bug hazards in Android , 2017, Empirical Software Engineering.

[41]  A. Hamou-Lhadj,et al.  A bug reproduction approach based on directed model checking and crash traces , 2017, J. Softw. Evol. Process..