Context-aware and data-driven feedback generation for programming assignments

Recently, various techniques have been proposed to automatically provide personalized feedback on programming exercises. The cutting edge of which is the data-driven approaches that leverage a corpus of existing correct programs and repair incorrect submissions by using similar reference programs in the corpus. However, current data-driven techniques work under the strong assumption that the corpus contains a solution program that is close enough to the incorrect submission. In this paper, we present Cafe, a new data-driven approach for feedback generation that overcomes this limitation. Unlike existing approaches, Cafe uses a novel context-aware repair algorithm that can generate feedback even if the incorrect program differs significantly from the reference solutions. We implemented Cafe for OCaml and evaluated it with 4,211 real student programs. The results show that Cafe is able to repair 83 % of incorrect submissions, far outperforming existing approaches.

[1]  Rahul Gupta,et al.  Neural Attribution for Semantic Bug-Localization in Student Programs , 2019, NeurIPS.

[2]  Xiangyu Zhang,et al.  Apex: automatic programming assignment error explanation , 2016, OOPSLA.

[3]  Aditya V. Thakur,et al.  Path-based function embedding and its application to error-handling specification mining , 2018, ESEC/SIGSOFT FSE.

[4]  Sumit Gulwani,et al.  Feedback generation for performance problems in introductory programming assignments , 2014, SIGSOFT FSE.

[5]  Björn Hartmann,et al.  Writing Reusable Code Feedback at Scale with Mixed-Initiative Program Synthesis , 2017, L@S.

[6]  Sumit Gulwani,et al.  Semi-supervised verified feedback generation , 2016, SIGSOFT FSE.

[7]  Philip J. Guo,et al.  OverCode: visualizing variation in student solutions to programming problems at scale , 2014, ACM Trans. Comput. Hum. Interact..

[8]  Sumit Gulwani,et al.  Learning Syntactic Program Transformations from Examples , 2016, 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE).

[9]  Sumit Gulwani,et al.  Compilation Error Repair: For the Student Programs, From the Student Programs , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET).

[10]  Claire Le Goues,et al.  Static Automated Program Repair for Heap Properties , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[11]  Ben Leong,et al.  Re-Factoring Based Program Repair Applied to Programming Assignments , 2019, 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[12]  Amey Karkare,et al.  Characterizing the Pedagogical Benefits of Adaptive Feedback for Compilation Errors by Novice Programmers , 2020, 2020 IEEE/ACM 42nd International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET).

[13]  Hakjoo Oh,et al.  MemFix: static analysis-based repair of memory deallocation errors for C , 2018, ESEC/SIGSOFT FSE.

[14]  Xiangyu Zhang,et al.  SemCluster: clustering of imperative programming assignments based on quantitative semantic features , 2019, PLDI.

[15]  Min Zhou,et al.  IntPTI: Automatic integer error repair with proper-type inference , 2017, 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[16]  Hakjoo Oh,et al.  Automatic and scalable detection of logical errors in functional programming assignments , 2019, Proc. ACM Program. Lang..

[17]  Monperrus Martin.  Automatic Software Repair: a Bibliography , 2020 .

[18]  Lu Zhang,et al.  Safe Memory-Leak Fixing for C Programs , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[19]  Arnab Bhattacharya,et al.  Automatic Grading and Feedback using Program Repair for Introductory Programming Courses , 2017, ITiCSE.

[20]  Rishabh Singh,et al.  Search, align, and repair: data-driven feedback generation for introductory programming exercises , 2017, PLDI.

[21]  Claire Le Goues,et al.  Automatically finding patches using genetic programming , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[22]  Abhik Roychoudhury,et al.  Angelix: Scalable Multiline Program Patch Synthesis via Symbolic Analysis , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[23]  Carlos R. Rivero,et al.  Automated Personalized Feedback in Introductory Java Programming MOOCs , 2017, 2017 IEEE 33rd International Conference on Data Engineering (ICDE).

[24]  Westley Weimer,et al.  Patches as better bug reports , 2006, GPCE '06.

[25]  Rishabh Singh,et al.  Qlose: Program Repair with Quantitative Objectives , 2016, CAV.

[26]  Sumit Gulwani,et al.  Automated feedback generation for introductory programming assignments , 2012, PLDI.

[27]  Ke Wang,et al.  Dynamic Neural Program Embedding for Program Repair , 2017, ICLR.

[28]  Lars Grunske,et al.  Semantic Program Repair Using a Reference Implementation , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[29]  OverCode , 2015 .

[30]  Daniela Micucci,et al.  Automatic Software Repair: A Survey , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[31]  Armando Solar-Lezama,et al.  sk_p: a neural program corrector for MOOCs , 2016, SPLASH.

[32]  Shuvendu K. Lahiri,et al.  Code vectors: understanding programs through embedded abstracted symbolic traces , 2018, ESEC/SIGSOFT FSE.

[33]  Martin Monperrus,et al.  Automatic Software Repair , 2018, ACM Comput. Surv..

[34]  Hua Yan,et al.  VFix: Value-Flow-Guided Precise Program Repair for Null Pointer Dereferences , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE).

[35]  Jeongsoo Lee,et al.  SAVER: Scalable, Precise, and Safe Memory-Error Repair , 2020, 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE).

[36]  Swarat Chaudhuri,et al.  Learning to Grade Student Programs in a Massive Open Online Course , 2014, 2014 IEEE International Conference on Data Mining.

[37]  Sumit Gulwani,et al.  Automated clustering and program repair for introductory programming assignments , 2016, PLDI.

[38]  Alex Shaw,et al.  Automatically Fixing C Buffer Overflows Using Program Transformations , 2014, 2014 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks.

[39]  Mark Harman,et al.  SapFix: Automated End-to-End Repair at Scale , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP).

[40]  Westley Weimer,et al.  Type error feedback via analytic program repair , 2020, PLDI.

[41]  Fan Long,et al.  Automatic patch generation by learning correct code , 2016, POPL.

[42]  Leonidas J. Guibas,et al.  Codewebs: scalable homework search for massive open online programming courses , 2014, WWW.

[43]  Fan Long,et al.  Staged program repair with condition synthesis , 2015, ESEC/SIGSOFT FSE.

[44]  Jiachen Zhang,et al.  Precise Condition Synthesis for Program Repair , 2016, 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE).

[45]  Hakjoo Oh,et al.  Automatic diagnosis and correction of logical errors for functional programming assignments , 2018, Proc. ACM Program. Lang..

[46]  Flemming Nielson,et al.  Principles of Program Analysis , 1999, Springer Berlin Heidelberg.

[47]  Johannes Bader,et al.  Getafix: learning to fix bugs automatically , 2019, Proc. ACM Program. Lang..

[48]  Xiao Liu,et al.  Automatic Grading of Programming Assignments: An Approach Based on Formal Semantics , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET).

[49]  Hakjoo Oh,et al.  Precise and scalable points-to analysis via data-driven context tunneling , 2018, Proc. ACM Program. Lang..