Evaluating Feedback Tools in Introductory Programming Classes

This Research Full Paper presents a study on the evaluation of feedback tools in introductory programming classes. Recently, several tools have been proposed in order to provide guidance and help students overcome conceptual difficulties in programming education. Some tools leverage clustering algorithms and program repair techniques to automatically generate personalized hints for students’ incorrect programs. In contrast, some teachers choose to present students with program visualization tools to help them understand the dynamic execution of a source code. These tools are used to help students get correct solutions for programming assignments. However, due to limitations in assessments, it is still unclear how effective the feedback provided by these tools is. In this study, we analyzed the effectiveness of a tool for generating personalized hints and a tool for visualizing programs. To do so, we conducted a user study in which students, assisted by these tools, implemented solutions for three programming problems. Our results show that personalized hints can significantly reduce student’s effort to get correct solutions. In addition, personalized hints can provide students with an understanding of problem solving similar to when using test cases. However, students who used the program visualization tool got lower post-test performance than using other tools.

[1]  Kenneth R. Koedinger,et al.  Data-Driven Hint Generation in Vast Solution Spaces: a Self-Improving Python Programming Tutor , 2015, International Journal of Artificial Intelligence in Education.

[2]  Philip J. Guo.  Codeopticon: Real-Time, One-To-Many Human Tutoring for Computer Programming , 2015, UIST.

[3]  Lauri Malmi,et al.  A Review of Generic Program Visualization Systems for Introductory Programming Education , 2013, TOCE.

[4]  Michael J. Morgan,et al.  Learning challenges faced by novice programming students studying high level and low feedback concepts , 2007 .

[5]  Sumit Gulwani,et al.  Automated feedback generation for introductory programming assignments , 2012, PLDI.

[6]  Rishabh Singh,et al.  Search, align, and repair: data-driven feedback generation for introductory programming exercises , 2017, PLDI.

[7]  Kenneth R. Koedinger,et al.  A Response Time Model For Bottom-Out Hints as Worked Examples , 2008, EDM.

[8]  V. Shute.  Focus on Formative Feedback , 2007 .

[9]  Armando Solar-Lezama,et al.  sk_p: a neural program corrector for MOOCs , 2016, SPLASH.

[10]  John T. Stasko,et al.  Please address correspondence to , 2000 .

[11]  Sumit Gulwani,et al.  Automated clustering and program repair for introductory programming assignments , 2016, PLDI.

[12]  John R. Anderson,et al.  Locus of feedback control in computer-based tutoring: impact on learning rate, achievement and attitudes , 2001, CHI.

[13]  Sumit Gulwani,et al.  How Can Automatic Feedback Help Students Construct Automata? , 2015, ACM Trans. Comput. Hum. Interact..

[14]  Cristina Conati,et al.  An Analysis of Attention to Student - Adaptive Hints in an Educational Game , 2012, ITS.

[15]  Oscar Karnalim,et al.  The Use of Python Tutor on Programming Laboratory Session: Student Perspectives , 2017 .

[16]  Philip J. Guo.  Online python tutor: embeddable web-based program visualization for cs education , 2013, SIGCSE '13.

[17]  Erkki Sutinen,et al.  Visualizing programs with Jeliot 3 , 2004, AVI.

[18]  Bharat Jayaraman,et al.  Interactive visualization of Java programs , 2002, Proceedings IEEE 2002 Symposia on Human Centric Computing Languages and Environments.

[19]  B. Myers.  Debugging Reinvented: Asking and Answering Why and Why Not Questions about Program Behavior , 2008 .

[20]  Tapio Salakoski,et al.  VILLE: a language-independent program visualization tool , 2007 .

[21]  Björn Hartmann,et al.  Writing Reusable Code Feedback at Scale with Mixed-Initiative Program Synthesis , 2017, L@S.

[22]  Sumit Gulwani,et al.  Semi-supervised verified feedback generation , 2016, SIGSOFT FSE.

[23]  Rishabh Singh,et al.  Qlose: Program Repair with Quantitative Objectives , 2016, CAV.

[24]  Björn Hartmann,et al.  TraceDiff: Debugging unexpected code behavior using trace divergences , 2017, 2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[25]  Sumit Gulwani,et al.  Learning Syntactic Program Transformations from Examples , 2016, 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE).

[26]  Rob Miller,et al.  Addressing misconceptions about code with always-on programming visualizations , 2014, CHI.

[27]  Philip J. Guo,et al.  OverCode: visualizing variation in student solutions to programming problems at scale , 2014, ACM Trans. Comput. Hum. Interact..