A Survey of Test Based Automatic Program Repair

Testing and debugging have always been the most time-consuming parts of the software development procedure and require large amounts of human resources. When a bug is located, manually fixing it to repair the buggy program is still a difficult and laborious task for developers. Hence automatic program repair techniques, especially the test-based approaches, have drawn great attentions in recent years. Researchers have explored and proposed various novel methods and tools, pushing the idea closer to reality. In this paper, we systematically survey the work in mainstream of test-based program repair (TBR) and discuss the properties automatically generated patches should have. We classify the state-of-the-art approaches for TBR, and evaluate their strengths and weaknesses according to their functional mechanisms. Finally, we refer to some empirical results and propose four important issues, which are supposed to be critical and constructive in this research area.

[1]  Koushik Sen,et al.  Concolic testing , 2007, ASE.

[2]  Fan Long,et al.  Automatic patch generation by learning correct code , 2016, POPL.

[3]  Armando Solar-Lezama,et al.  Program sketching , 2012, International Journal on Software Tools for Technology Transfer.

[4]  Matias Martinez,et al.  Mining software repair models for reasoning on the search space of automated program fixing , 2013, Empirical Software Engineering.

[5]  Claire Le Goues,et al.  Automatically finding patches using genetic programming , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[6]  Corina S. Pasareanu,et al.  A survey of new trends in symbolic execution for software testing and analysis , 2009, International Journal on Software Tools for Technology Transfer.

[7]  Rajiv Gupta,et al.  BugFix: A learning-based tool to assist developers in fixing bugs , 2009, 2009 IEEE 17th International Conference on Program Comprehension.

[8]  Mary Jean Harrold,et al.  Empirical evaluation of the tarantula automatic fault-localization technique , 2005, ASE.

[9]  Lionel C. Briand,et al.  A practical guide for using statistical tests to assess randomized algorithms in software engineering , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[10]  Westley Weimer,et al.  Leveraging program equivalence for adaptive program repair: Models and first results , 2013, 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[11]  Andreas Zeller,et al.  Mining object behavior with ADABU , 2006, WODA '06.

[12]  Natarajan Shankar,et al.  A Tutorial on Satisfiability Modulo Theories , 2007, CAV.

[13]  Charles Zhang,et al.  Axis: Automatically fixing atomicity violations through solving control constraints , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[14]  Andreas Zeller,et al.  Generating Fixes from Object Behavior Anomalies , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[15]  Yuhua Qi,et al.  The strength of random search on automated program repair , 2014, ICSE.

[16]  Ron Weber,et al.  Some factors affecting program repair maintenance: an empirical study , 1983, CACM.

[17]  Lee Naish,et al.  Spectral Debugging with Weights and Incremental Ranking , 2009, 2009 16th Asia-Pacific Software Engineering Conference.

[18]  Albert Endres,et al.  An analysis of errors and their causes in system programs , 1975, IEEE Transactions on Software Engineering.

[19]  Abhik Roychoudhury,et al.  Codeflaws: A Programming Competition Benchmark for Evaluating Automated Program Repair Tools , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C).

[20]  Bertrand Meyer,et al.  Automated Program Repair in an Integrated Development Environment , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[21]  Sarfraz Khurshid,et al.  Symbolic execution for software testing in practice: preliminary assessment , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[22]  Sunghun Kim,et al.  Automatically generated patches as debugging aids: a human study , 2014, SIGSOFT FSE.

[23]  Frank Tip,et al.  Automated repair of HTML generation errors in PHP applications using string constraint solving , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[24]  Claire Le Goues,et al.  A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[25]  Brendan Murphy,et al.  The Art of Testing Less without Sacrificing Quality , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[26]  W. Eric Wong,et al.  Using Mutation to Automatically Suggest Fixes for Faulty Programs , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[27]  Bertrand Meyer,et al.  Automatic Program Repair by Fixing Contracts , 2014, FASE.

[28]  Abhik Roychoudhury,et al.  DirectFix: Looking for Simple Program Repairs , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[29]  Elaine J. Weyuker,et al.  Predicting the location and number of faults in large software systems , 2005, IEEE Transactions on Software Engineering.

[30]  Yuriy Brun,et al.  The ManyBugs and IntroClass Benchmarks for Automated Repair of C Programs , 2015, IEEE Transactions on Software Engineering.

[31]  Kathryn T. Stolee,et al.  Repairing Programs with Semantic Code Search , 2015 .

[32]  Fan Long,et al.  An analysis of patch plausibility and correctness for generate-and-validate patch generation systems , 2015, ISSTA.

[33]  Philip Koopman,et al.  Elements of the Self-Healing System Problem Space , 2003 .

[34]  Martin Monperrus,et al.  A critical review of "automatic patch generation learned from human-written patches": essay on the problem statement and the evaluation of automatic software repair , 2014, ICSE.

[35]  Yuhua Qi,et al.  Efficient Automated Program Repair through Fault-Recorded Testing Prioritization , 2013, 2013 IEEE International Conference on Software Maintenance.

[36]  Shinji Kusumoto,et al.  Toward Developer-like Automated Program Repair — Modification Comparisons between GenProg and Developers , 2016, 2016 23rd Asia-Pacific Software Engineering Conference (APSEC).

[37]  Abhik Roychoudhury,et al.  relifix: Automated Repair of Software Regressions , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[38]  Clark W. Barrett,et al.  The SMT-LIB Standard Version 2.0 , 2010 .

[39]  Lu Zhang,et al.  Identifying Patch Correctness in Test-Based Automatic Program Repair , 2017, arXiv.org.

[40]  Claire Le Goues,et al.  Current challenges in automatic software repair , 2013, Software Quality Journal.

[41]  David Lo,et al.  Empirical Study on Synthesis Engines for Semantics-Based Program Repair , 2016, 2016 IEEE International Conference on Software Maintenance and Evolution (ICSME).

[42]  Martin Monperrus,et al.  Automatic repair of buggy if conditions and missing preconditions with SMT , 2014, CSTVA 2014.

[43]  Ali Mesbah,et al.  Vejovis: suggesting fixes for JavaScript faults , 2014, ICSE.

[44]  Claire Le Goues,et al.  Using Execution Paths to Evolve Software Patches , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[45]  Viktor Kuncak,et al.  On Deductive Program Repair in Leon , 2015 .

[46]  Michael D. Ernst,et al.  Automatically patching errors in deployed software , 2009, SOSP '09.

[47]  Roderick Bloem,et al.  Automated error localization and correction for imperative programs , 2011, 2011 Formal Methods in Computer-Aided Design (FMCAD).

[48]  Dawei Qi,et al.  SemFix: Program repair via semantic analysis , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[49]  Xin Yao,et al.  Coevolving programs and unit tests from their specification , 2007, ASE.

[50]  Andreas Zeller,et al.  Why Programs Fail: A Guide to Systematic Debugging , 2005 .

[51]  Baowen Xu,et al.  A theoretical analysis of the risk evaluation formulas for spectrum-based fault localization , 2013, TSEM.

[52]  Claire Le Goues,et al.  A genetic programming approach to automated software repair , 2009, GECCO.

[53]  Joost Visser,et al.  A Practical Model for Measuring Maintainability , 2007, 6th International Conference on the Quality of Information and Communications Technology (QUATIC 2007).

[54]  Jiachen Zhang,et al.  Precise Condition Synthesis for Program Repair , 2016, 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE).

[55]  Neelam Gupta,et al.  Automated Debugging Using Path-Based Weakest Preconditions , 2004, FASE.

[56]  Sunghun Kim,et al.  Toward an understanding of bug fix patterns , 2009, Empirical Software Engineering.

[57]  Victor R. Basili,et al.  Software errors and complexity: an empirical investigation , 1993 .

[58]  Westley Weimer,et al.  A human study of patch maintainability , 2012, ISSTA 2012.

[59]  Brent Hailpern,et al.  Software debugging, testing, and verification , 2002, IBM Syst. J..

[60]  Shan Lu,et al.  CARAMEL: Detecting and Fixing Performance Problems That Have Non-Intrusive Fixes , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[61]  Liam Schramm,et al.  Improving performance of automatic program repair using learned heuristics , 2017, ESEC/SIGSOFT FSE.

[62]  Zuohua Ding,et al.  A Revisit of the Integration of Metamorphic Testing and Test Suite Based Automated Program Repair , 2017, 2017 IEEE/ACM 2nd International Workshop on Metamorphic Testing (MET).

[63]  Matias Martinez,et al.  Automatic Repair of Real Bugs: An Experience Report on the Defects4J Dataset , 2015, ArXiv.

[64]  Alexey Zhikhartsev,et al.  Better test cases for better automated program repair , 2017, ESEC/SIGSOFT FSE.

[65]  Nikolaj Bjørner,et al.  Satisfiability modulo theories , 2011, Commun. ACM.

[66]  Fan Long,et al.  Staged program repair with condition synthesis , 2015, ESEC/SIGSOFT FSE.

[67]  Matias Martinez,et al.  Mining Repair Actions for Guiding Automated Program Fixing , 2012 .

[68]  Yuriy Brun,et al.  Is the cure worse than the disease? overfitting in automated program repair , 2015, ESEC/SIGSOFT FSE.

[69]  Gail C. Murphy,et al.  Coping with an open bug repository , 2005, eclipse '05.

[70]  Claire Le Goues,et al.  GenProg: A Generic Method for Automatic Software Repair , 2012, IEEE Transactions on Software Engineering.

[71]  Shin Yoo,et al.  Ask the Mutants: Mutating Faulty Programs for Fault Localization , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[72]  James M. Bieman,et al.  An Assessment of the Quality of Automated Program Operator Repair , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[73]  Xin Yao,et al.  A novel co-evolutionary approach to automatic software bug fixing , 2008, 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence).

[74]  Jaechang Nam,et al.  Automatic patch generation learned from human-written patches , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[75]  Monperrus Martin.  Automatic Software Repair: a Bibliography , 2020 .

[76]  A.J.C. van Gemund,et al.  On the Accuracy of Spectrum-based Fault Localization , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[77]  Abhik Roychoudhury,et al.  Angelix: Scalable Multiline Program Patch Synthesis via Symbolic Analysis , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).