Type error feedback via analytic program repair

We introduce Analytic Program Repair, a data-driven strategy for providing feedback for type-errors via repairs for the erroneous program. Our strategy is based on insight that similar errors have similar repairs. Thus, we show how to use a training dataset of pairs of ill-typed programs and their fixed versions to: (1) learn a collection of candidate repair templates by abstracting and partitioning the edits made in the training set into a representative set of templates; (2) predict the appropriate template from a given error, by training multi-class classifiers on the repair templates used in the training set; (3) synthesize a concrete repair from the template by enumerating and ranking correct (e.g. well-typed) terms matching the predicted template. We have implemented our approach in Rite: a type error reporting tool for OCaml programs. We present an evaluation of the accuracy and efficiency of Rite on a corpus of 4,500 ill-typed Ocaml programs drawn from two instances of an introductory programming course, and a user-study of the quality of the generated error messages that shows the locations and final repair quality to be better than the state-of-the-art tool in a statistically-significant manner.

[1]  Earl T. Barr,et al.  Learning Python Code Suggestion with a Sparse Pointer Network , 2016, ArXiv.

[2]  Seemanta Saha,et al.  Harnessing Evolution for Multi-Hunk Program Repair , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE).

[3]  Rishabh Singh,et al.  Search, align, and repair: data-driven feedback generation for introductory programming exercises , 2017, PLDI.

[4]  David Lo,et al.  Practitioners' expectations on automated fault localization , 2016, ISSTA.

[5]  Matias Martinez,et al.  Automatically Extracting Instances of Code Change Patterns with AST Analysis , 2013, 2013 IEEE International Conference on Software Maintenance.

[6]  Fan Long,et al.  Automatic patch generation by learning correct code , 2016, POPL.

[7]  Sumit Gulwani,et al.  Compilation Error Repair: For the Student Programs, From the Student Programs , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET).

[8]  Fairouz Kamareddine,et al.  Skalpel: A Type Error Slicer for Standard ML , 2014, LSFA.

[9]  Björn Hartmann,et al.  Writing Reusable Code Feedback at Scale with Mixed-Initiative Program Synthesis , 2017, L@S.

[10]  Mitchell Wand.  Finding the source of type errors , 1986, POPL '86.

[11]  P MunsonJonathan,et al.  Analyzing novice programmers' response to compiler error messages , 2016 .

[12]  T. B. Dinesh,et al.  Centrum Voor Wiskunde En Informatica Reportrapport a Slicing-based Approach for Locating Type Errors a Slicing-based Approach for Locating Type Errors , 2022 .

[13]  Martin Erwig,et al.  Counter-factual typing for debugging type errors , 2014, POPL.

[14]  D. Ruppert.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2004 .

[15]  Loris D'Antoni,et al.  Learning Quick Fixes from Code Repositories , 2018, SBES.

[16]  Charu C. Aggarwal,et al.  Neural Networks and Deep Learning , 2018, Springer International Publishing.

[17]  Danfeng Zhang,et al.  Toward general diagnosis of static errors , 2014, POPL.

[18]  Kamalika Chaudhuri,et al.  Learning to blame: localizing novice type errors with data-driven diagnosis , 2017, Proc. ACM Program. Lang..

[19]  Jordi Levy,et al.  Anti-unification for Unranked Terms and Hedges , 2013, Journal of Automated Reasoning.

[20]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[21]  Matias Martinez,et al.  Mining software repair models for reasoning on the search space of automated program fixing , 2013, Empirical Software Engineering.

[22]  Robert Hieb,et al.  The Revised Report on the Syntactic Theories of Sequential Control and State , 1992, Theor. Comput. Sci..

[23]  Johannes Bader,et al.  Getafix: learning to fix bugs automatically , 2019, Proc. ACM Program. Lang..

[24]  Andres Löh,et al.  Type-safe diff for families of datatypes , 2009, WGP '09.

[25]  Manu Sridharan,et al.  Tech Report : A Practical Framework for Type Inference Error Explanation , 2016 .

[26]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[27]  Christian Haack,et al.  Type error slicing in implicitly typed higher-order languages , 2003, Sci. Comput. Program..

[28]  Zhendong Su,et al.  A study of the uniqueness of source code , 2010, FSE '10.

[29]  Alessandro Orso,et al.  Are automated debugging techniques actually helping programmers? , 2011, ISSTA '11.

[30]  Zvonimir Pavlinovic,et al.  Finding minimum type error sources , 2014, Software Engineering & Management.

[31]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[32]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[33]  Le Song,et al.  Hoppity: Learning Graph Transformations to Detect and Fix Bugs in Programs , 2020, ICLR.

[34]  Premkumar T. Devanbu,et al.  On the naturalness of software , 2016, Commun. ACM.

[35]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[36]  Marc Brockschmidt,et al.  Learning to Represent Programs with Graphs , 2017, ICLR.

[37]  Sumit Gulwani,et al.  Automated feedback generation for introductory programming assignments , 2012, PLDI.

[38]  Dan Grossman,et al.  Searching for type-error messages , 2007, PLDI '07.

[39]  Jaechang Nam,et al.  Automatic patch generation learned from human-written patches , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[40]  Sumit Gulwani,et al.  Automated clustering and program repair for introductory programming assignments , 2016, PLDI.

[41]  Westley Weimer,et al.  Dynamic witnesses for static type errors (or, ill-typed programs usually go wrong) , 2016, Journal of Functional Programming.

[42]  GulwaniSumit,et al.  Automated feedback generation for introductory programming assignments , 2013 .

[43]  Hakjoo Oh,et al.  Automatic and scalable detection of logical errors in functional programming assignments , 2019, Proc. ACM Program. Lang..