The PyramidSnapshot Challenge: Understanding Student Process from Visual Output of Programs

In the ideal CS1 classroom, we should understand programming process---how student code evolves over time. However, for graphics-based programming assignments, the task of understanding and grading final solutions, let alone thousands of intermediate steps, is incredibly labor-intensive. In this work, we present a challenge, a dataset, and a promising first solution to autonomously use image output to identify functional, intermediate stages of a student solution. By using computer vision techniques to associate visual output of intermediate student code with functional progress, we supplement a lot of the teacher labor associated with understanding graphics-based, open-ended assignments. We hope our publication of the dataset used in our case study sparks discussion in the community on how to analyze programs with visual program output.

[1]  Gertrude W. Abramson.  Dann, W., Cooper, S. & Pausch, R. (2006). Learning to program with Alice. Upper Saddle River, NJ: Prentice-Hall , 2008 .

[2]  Matthew C. Jadud,et al.  Methods and tools for exploring novice compilation behaviour , 2006, ICER '06.

[3]  Tao Wang,et al.  Convolutional Neural Networks over Tree Structures for Programming Language Processing , 2014, AAAI.

[4]  Randy Pausch,et al.  Learning To Program With Alice , 2004 .

[5]  James J. Kuffner,et al.  Session details: Computer games and CS education: why and how , 2005, SIGCSE '05.

[6]  Paulo Blikstein,et al.  Modeling how students learn to program , 2012, SIGCSE '12.

[7]  Eric Rosenbaum,et al.  Scratch: programming for all , 2009, Commun. ACM.

[8]  David Hovemeyer,et al.  Control-Flow-Only Abstract Syntax Trees for Analyzing Students' Programming Progress , 2016, ICER.

[9]  Lucas Layman,et al.  Note to self: make assignments meaningful , 2007, SIGCSE '07.

[10]  Nick McKeown,et al.  TMOSS: Using Intermediate Assignment Work to Understand Excessive Collaboration in Large Classes , 2018, SIGCSE.

[11]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[12]  Ke Wang,et al.  Dynamic Neural Program Embedding for Program Repair , 2017, ICLR.

[13]  Daniel D. Garcia,et al.  ACES: Automatic Evaluation of Coding Style , 2014 .

[14]  Lisa Yan,et al.  Pensieve: Feedback on Coding Process for Novices , 2019, SIGCSE.

[15]  Eric Roberts,et al.  Strategies for encouraging individual achievement in introductory computer science courses , 2000, SIGCSE '00.

[16]  Marcelo Worsley,et al.  Programming Pluralism: Using Learning Analytics to Detect Patterns in the Learning of Computer Programming , 2014, Journal of the Learning Sciences.

[17]  Raymond Lister,et al.  Exploring Machine Learning Methods to Automatically Identify Students in Need of Assistance , 2015, ICER.

[18]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[19]  Leonidas J. Guibas,et al.  Learning Program Embeddings to Propagate Feedback on Student Code , 2015, ICML.

[20]  Brett A. Becker.  A New Metric to Quantify Repeated Compiler Errors for Novice Programmers , 2016, ITiCSE.

[21]  Chris Piech,et al.  Deep Knowledge Tracing On Programming Exercises , 2017, L@S.

[22]  David S. Janzen,et al.  Test-driven learning: intrinsic integration of testing into the CS/SE curriculum , 2006, SIGCSE '06.

[23]  S. Piantadosi.  Zipf’s word frequency law in natural language: A critical review and future directions , 2014, Psychonomic Bulletin & Review.

[24]  Colleen M. Lewis.  The importance of students' attention to program state: a case study of debugging behavior , 2012, ICER '12.

[25]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[26]  Leonidas J. Guibas,et al.  Syntactic and Functional Variability of a Million Code Submissions in a Machine Learning MOOC , 2013, AIED Workshops.

[27]  Christopher D. Hundhausen,et al.  The Normalized Programming State Model: Predicting Student Performance in Computing Courses Based on Programming Behavior , 2015, ICER.

[28]  Mark Guzdial,et al.  Subgoals, Context, and Worked Examples in Learning Computing Problem Solving , 2015, ICER.

[29]  Vijay Vasudevan,et al.  Learning Transferable Architectures for Scalable Image Recognition , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[30]  S. Turkle,et al.  Epistemological Pluralism: Styles and Voices within the Computer Culture , 1990, Signs: Journal of Women in Culture and Society.

[31]  Taylor Martin,et al.  Using Learning Analytics to Understand the Learning Pathways of Novice Programmers , 2013 .

[32]  Barbara Hammer,et al.  Domain-Independent Proximity Measures in Intelligent Tutoring Systems , 2013, EDM.

[33]  Stephen H. Edwards,et al.  Web-CAT: automatically grading programming assignments , 2008, ITiCSE.

[34]  Paulo Blikstein,et al.  Using learning analytics to assess students' behavior in open-ended programming tasks , 2011, LAK.