Characterizing the Pedagogical Benefits of Adaptive Feedback for Compilation Errors by Novice Programmers

Can automated adaptive feedback for correcting erroneous programs help novice programmers learn to code better? In a large-scale experiment, we compare student performance when tutored by human tutors, and when receiving automated adaptive feedback. The automated feedback was designed using one of two well-known instructional principles: (i) presenting the correct solution for the immediate problem, or (ii) presenting generated examples or analogies that guide towards the correct solution. We report empirical results from a large-scale (N = 480,10,000+ person hour) experiment assessing the efficacy of these automated compilation-error feedback tools. Using the survival analysis on error rates of students measured over seven weeks, we found that automated feedback allows students to resolve errors in their code more efficiently than students receiving manual feedback. However, we also found that this advantage is primarily logistical and not conceptual; the performance benefit seen during lab assignments disappeared during exams wherein feedback of any kind was withdrawn. We further found that the performance advantage of automated feedback over human tutors increases with problem complexity, and that feedback via example and specific repair have distinct, non-overlapping relative advantages for different categories of programming errors. Our results offer a clear and granular delimitation of the pedagogical benefits of automated feedback in teaching programming to novices.

[1]  E. Kaplan,et al.  Nonparametric Estimation from Incomplete Observations , 1958 .

[2]  James J. Horning,et al.  What the Compiler Should Tell the User , 1976, Compiler Construction.

[3]  Rupert G. Miller,et al.  Survival Analysis , 2022, The SAGE Encyclopedia of Research Design.

[4]  Ben Shneiderman,et al.  Designing computer system messages , 1982, CACM.

[5]  H. Walberg,et al.  Instructional Effects of Cues, Participation, and Corrective Feedback: A Quantitative Synthesis , 1982 .

[6]  John R. Anderson,et al.  Locus of feedback control in computer-based tutoring: impact on learning rate, achievement and attitudes , 2001, CHI.

[7]  Judy Kay,et al.  The Neglected Battle Fields of Syntax Errors , 2003, ACE.

[8]  K. Morgan‐Short,et al.  Positive Evidence Versus Explicit Rule Presentation and Explicit Negative Feedback: A Computer-Assisted Study. , 2004 .

[9]  Vikram S. Adve,et al.  LLVM: a compilation framework for lifelong program analysis & transformation , 2004, International Symposium on Code Generation and Optimization, 2004. CGO 2004..

[10]  Bertrand Meyer,et al.  Compiler error messages: what can help novices? , 2008, SIGCSE '08.

[11]  Sue Fitzgerald,et al.  Debugging: a review of the literature from an educational perspective , 2008, Comput. Sci. Educ..

[12]  Scott R. Klemmer,et al.  What would other programmers do: suggesting solutions to error messages , 2010, CHI.

[13]  V. Javier Traver,et al.  On Compiler Error Messages: What They Say and What They Mean , 2010, Adv. Hum. Comput. Interact..

[14]  Albert T. Corbett,et al.  The Knowledge-Learning-Instruction Framework: Bridging the Science-Practice Chasm to Enhance Robust Student Learning , 2012, Cogn. Sci..

[15]  Frederick W. B. Li,et al.  BlueFix: Using Crowd-Sourced Feedback to Support Programming Students in Error Diagnosis and Repair , 2012, ICWL.

[16]  Ewan D. Tempero,et al.  All syntax errors are not equal , 2012, ITiCSE '12.

[17]  GulwaniSumit,et al.  Automated feedback generation for introductory programming assignments , 2013 .

[18]  Sumit Gulwani,et al.  Automated feedback generation for introductory programming assignments , 2012, ACM-SIGPLAN Symposium on Programming Language Design and Implementation.

[19]  Alexander Renkl,et al.  Toward an Instructionally Oriented Theory of Example-Based Learning , 2014, Cogn. Sci..

[20]  Noah D. Goodman,et al.  A rational account of pedagogical reasoning: Teaching by, and learning from, examples , 2014, Cognitive Psychology.

[21]  Andrew Luxton-Reilly,et al.  Enhancing syntax error messages appears ineffectual , 2014, ITiCSE '14.

[22]  OverCode , 2015 .

[23]  Philip J. Guo.  Codeopticon: Real-Time, One-To-Many Human Tutoring for Computer Programming , 2015, UIST.

[24]  Philip J. Guo,et al.  OverCode: visualizing variation in student solutions to programming problems at scale , 2014, ACM Trans. Comput. Hum. Interact..

[25]  Sumit Gulwani,et al.  How Can Automatic Feedback Help Students Construct Automata? , 2015, ACM Trans. Comput. Hum. Interact..

[26]  Armando Solar-Lezama,et al.  sk_p: a neural program corrector for MOOCs , 2016, SPLASH.

[27]  Rishabh Singh,et al.  Automated Correction for Syntax Errors in Programming Assignments using Recurrent Neural Networks , 2016, ArXiv.

[28]  Sumit Gulwani,et al.  Prutor: A System for Tutoring CS1 and Collecting Student Programs for Analysis , 2016, ArXiv.

[29]  Brett A. Becker.  An Effective Approach to Enhancing Compiler Error Messages , 2016, SIGCSE.

[30]  John Homer,et al.  On Novices' Interaction with Compiler Error Messages: A Human Factors Approach , 2017, ICER.

[31]  Rahul Gupta,et al.  DeepFix: Fixing Common C Language Errors by Deep Learning , 2017, AAAI.

[32]  Amey Karkare,et al.  A feasibility study of using automated program repair for introductory programming assignments , 2017, ESEC/SIGSOFT FSE.

[33]  Phitchaya Mangpo Phothilimthana,et al.  High-Coverage Hint Generation for Massive Courses: Do Automated Hints Help CS1 Students? , 2017, ITiCSE.

[34]  John Homer,et al.  Do Enhanced Compiler Error Messages Help Students?: Results Inconclusive. , 2017, SIGCSE.

[35]  Sumit Gulwani,et al.  Learning Syntactic Program Transformations from Examples , 2016, 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE).

[36]  Arnab Bhattacharya,et al.  Automatic Grading and Feedback using Program Repair for Introductory Programming Courses , 2017, ITiCSE.

[37]  Emerson R. Murphy-Hill,et al.  How should compilers explain problems to developers? , 2018, ESEC/SIGSOFT FSE.

[38]  Sumit Gulwani,et al.  Automated clustering and program repair for introductory programming assignments , 2016, PLDI.

[39]  Sumit Gulwani,et al.  Compilation Error Repair: For the Student Programs, From the Student Programs , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET).

[40]  Brett A. Becker,et al.  The Effects of Enhanced Compiler Error Messages on a Syntax Error Debugging Test , 2018, SIGCSE.

[41]  Rishabh Singh,et al.  Search, align, and repair: data-driven feedback generation for introductory programming exercises , 2017, PLDI.

[42]  SuZhendong,et al.  Search, align, and repair: data-driven feedback generation for introductory programming exercises , 2018 .

[43]  Amey Karkare,et al.  Unexpected Tokens: A Review of Programming Error Messages and Design Guidelines for the Future , 2019, ITiCSE.

[44]  Amey Karkare,et al.  Targeted Example Generation for Compilation Errors , 2019, 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[45]  Ben Leong,et al.  Re-Factoring Based Program Repair Applied to Programming Assignments , 2019, 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[46]  Rahul Gupta,et al.  Deep Reinforcement Learning for Syntactic Error Repair in Student Programs , 2019, AAAI.

[47]  Christoph Treude,et al.  Enhancing Python Compiler Error Messages via Stack , 2019, 2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM).