Experience Report: How Effective is Automated Program Repair for Industrial Software?

Recent advances in automated program repair (APR) have widely caught the attention of industrial developers as a way of reducing debugging costs. While hundreds of studies have evaluated the effectiveness of APR on open-source software, industrial case studies on APR have been rarely reported; it is still unclear whether APR can work well for industrial software. This paper reports our experience applying a state-of-the-art APR technique, ELIXIR, to large industrial software consisting of 150+ Java projects and 13 years of development histories. It provides lessons learned and recommendations regarding obstacles to the industrial use of current APR: low recall (7.7%), lack of bug-exposing tests (90%), low success rate (10%), among others. We also report the preliminary results of our ongoing improvement of ELIXIR. With some simple enhancements, the success rate of repair has been greatly improved by up to 40%.

[1]  Hiroaki Yoshida,et al.  Phoenix: automated data-driven synthesis of repairs for static analysis violations , 2019, ESEC/SIGSOFT FSE.

[2]  Tegawendé F. Bissyandé,et al.  TBar: revisiting template-based automated program repair , 2019, ISSTA.

[3]  Hongyu Zhang,et al.  Shaping program repair space with existing patches and similar code , 2018, ISSTA.

[4]  Hiroaki Yoshida,et al.  Elixir: Effective object-oriented program repair , 2017, 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[5]  Jacques Klein,et al.  iFixR: bug report driven program repair , 2019, ESEC/SIGSOFT FSE.

[6]  Shinji Kusumoto,et al.  Toward Introducing Automated Program Repair Techniques to Industrial Software Development , 2018, 2018 IEEE/ACM 26th International Conference on Program Comprehension (ICPC).

[7]  Seemanta Saha,et al.  Harnessing Evolution for Multi-Hunk Program Repair , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE).

[8]  Denys Poshyvanyk,et al.  SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair , 2018, IEEE Transactions on Software Engineering.

[9]  Yuriy Brun,et al.  The plastic surgery hypothesis , 2014, SIGSOFT FSE.

[10]  Matias Martinez,et al.  Alleviating patch overfitting with automatic test generation: a study of feasibility and effectiveness for the Nopol repair system , 2018, Empirical Software Engineering.

[11]  Matias Martinez,et al.  ASTOR: a program repair library for Java (demo) , 2016, ISSTA.

[12]  Michael D. Ernst,et al.  Defects4J: a database of existing faults to enable controlled testing studies for Java programs , 2014, ISSTA 2014.

[13]  Abhik Roychoudhury,et al.  Angelix: Scalable Multiline Program Patch Synthesis via Symbolic Analysis , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[14]  Daniela Micucci,et al.  Automatic Software Repair: A Survey , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[15]  Ming Wen,et al.  Context-Aware Patch Generation for Better Automated Program Repair , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[16]  Johannes Bader,et al.  Getafix: learning to fix bugs automatically , 2019, Proc. ACM Program. Lang..

[17]  Rui Abreu,et al.  Empirical review of Java program repair tools: a large-scale experiment on 2,141 bugs and 23,551 repair attempts , 2019, ESEC/SIGSOFT FSE.

[18]  Martin Monperrus,et al.  Nopol: Automatic Repair of Conditional Statement Bugs in Java Programs , 2018, IEEE Transactions on Software Engineering.

[19]  Mark Harman,et al.  SapFix: Automated End-to-End Repair at Scale , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP).

[20]  Claire Le Goues,et al.  A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each , 2012, 2012 34th International Conference on Software Engineering (ICSE).