The strength of random search on automated program repair

Automated program repair recently received considerable attentions, and many techniques on this research area have been proposed. Among them, two genetic-programming-based techniques, GenProg and Par, have shown the promising results. In particular, GenProg has been used as the baseline technique to check the repair effectiveness of new techniques in much literature. Although GenProg and Par have shown their strong ability of fixing real-life bugs in nontrivial programs, to what extent GenProg and Par can benefit from genetic programming, used by them to guide the patch search process, is still unknown. To address the question, we present a new automated repair technique using random search, which is commonly considered much simpler than genetic programming, and implement a prototype tool called RSRepair. Experiment on 7 programs with 24 versions shipping with real-life bugs suggests that RSRepair, in most cases (23/24), outperforms GenProg in terms of both repair effectiveness (requiring fewer patch trials) and efficiency (requiring fewer test case executions), justifying the stronger strength of random search over genetic programming. According to experimental results, we suggest that every proposed technique using optimization algorithm should check its effectiveness by comparing it with random search.

[1]  Yuanyuan Zhang,et al.  Search-based software engineering: Trends, techniques and applications , 2012, CSUR.

[2]  Yuhua Qi,et al.  Efficient Automated Program Repair through Fault-Recorded Testing Prioritization , 2013, 2013 IEEE International Conference on Software Maintenance.

[3]  Baowen Xu,et al.  A theoretical analysis of the risk evaluation formulas for spectrum-based fault localization , 2013, TSEM.

[4]  Claire Le Goues,et al.  Current challenges in automatic software repair , 2013, Software Quality Journal.

[5]  Yuhua Qi,et al.  Does Genetic Programming Work Well on Automated Program Repair? , 2013, 2013 International Conference on Computational and Information Sciences.

[6]  Westley Weimer,et al.  A human study of patch maintainability , 2012, ISSTA 2012.

[7]  Bertrand Meyer,et al.  Code-based automated program fixing , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[8]  F. Wilcoxon.  Individual Comparisons by Ranking Methods , 1945 .

[9]  Bin Gu,et al.  More efficient automatic repair of large-scale programs using weak recompilation , 2012, Science China Information Sciences.

[10]  Jaechang Nam,et al.  Automatic patch generation learned from human-written patches , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[11]  Gordon Fraser,et al.  Search Based Software Engineering , 2012, Lecture Notes in Computer Science.

[12]  HarmanMark.  Automated patching techniques , 2010 .

[13]  Claire Le Goues,et al.  Automatically finding patches using genetic programming , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[14]  Andreas Zeller,et al.  Automated Fixing of Programs with Contracts , 2014 .

[15]  John A. Clark,et al.  Efficient Software Verification: Statistical Testing Using Automated Search , 2010, IEEE Transactions on Software Engineering.

[16]  Alessandro Orso,et al.  F3: fault localization for field failures , 2013, ISSTA.

[17]  Charles Zhang,et al.  Axis: Automatically fixing atomicity violations through solving control constraints , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[18]  Andreas Zeller,et al.  Generating Fixes from Object Behavior Anomalies , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[19]  Bogdan Korel,et al.  Application of system models in regression test suite prioritization , 2008, 2008 IEEE International Conference on Software Maintenance.

[20]  Westley Weimer,et al.  Leveraging program equivalence for adaptive program repair: Models and first results , 2013, 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[21]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[22]  Gregg Rothermel,et al.  Empirical studies of test case prioritization in a JUnit testing environment , 2004, 15th International Symposium on Software Reliability Engineering.

[23]  Claire Le Goues,et al.  GenProg: A Generic Method for Automatic Software Repair , 2012, IEEE Transactions on Software Engineering.

[24]  Yuhua Qi,et al.  Using automated program repair for evaluating the effectiveness of fault localization techniques , 2013, ISSTA.

[25]  Andrea Arcuri,et al.  On the automation of fixing software bugs , 2008, ICSE Companion '08.

[26]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[27]  Claire Le Goues.  Automatic Program Repair Using Genetic Programming , 2013 .

[28]  Martin Monperrus,et al.  A critical review of "automatic patch generation learned from human-written patches": essay on the problem statement and the evaluation of automatic software repair , 2014, ICSE.

[29]  Gregg Rothermel,et al.  Test Case Prioritization: A Family of Empirical Studies , 2002, IEEE Trans. Software Eng..

[30]  W. Eric Wong,et al.  Using Mutation to Automatically Suggest Fixes for Faulty Programs , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[31]  Mark Harman,et al.  Search Based Software Engineering: Techniques, Taxonomy, Tutorial , 2010, LASER Summer School.

[32]  Claire Le Goues,et al.  Automatic program repair with evolutionary computation , 2010, Commun. ACM.

[33]  Roderick Bloem,et al.  Repair of Boolean Programs with an Application to C , 2006, CAV.

[34]  Dawei Qi,et al.  SemFix: Program repair via semantic analysis , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[35]  A. Vargha,et al.  A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong , 2000 .

[36]  Frank Tip,et al.  Automated repair of HTML generation errors in PHP applications using string constraint solving , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[37]  Lionel C. Briand,et al.  A practical guide for using statistical tests to assess randomized algorithms in software engineering , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[38]  Claire Le Goues,et al.  A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[39]  Claire Le Goues,et al.  Representations and operators for improving evolutionary software repair , 2012, GECCO '12.

[40]  Shan Lu,et al.  Automated atomicity-violation fixing , 2011, PLDI '11.

[41]  Claire Le Goues,et al.  Designing better fitness functions for automated program repair , 2010, GECCO '10.

[42]  Bradley Alexander,et al.  Evolving patches for software repair , 2011, GECCO '11.

[43]  Andrea Arcuri,et al.  Evolutionary repair of faulty software , 2011, Appl. Soft Comput..

[44]  Tao Xie,et al.  Time-aware test-case prioritization using integer linear programming , 2009, ISSTA.

[45]  Xin Yao,et al.  A novel co-evolutionary approach to automatic software bug fixing , 2008, 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence).