Educational Data Mining and Learning Analytics in Programming: Literature Review and Case Studies

Educational data mining and learning analytics promise better understanding of student behavior and knowledge, as well as new information on the tacit factors that contribute to student actions. This knowledge can be used to inform decisions related to course and tool design and pedagogy, and to further engage students and guide those at risk of failure. This working group report provides an overview of the body of knowledge regarding the use of educational data mining and learning analytics focused on the teaching and learning of programming. In a literature survey on mining students' programming processes for 2005-2015, we observe a significant increase in work related to the field. However, the majority of the studies focus on simplistic metric analysis and are conducted within a single institution and a single course. This indicates the existence of further avenues of research and a critical need for validation and replication to better understand the various contributing factors and the reasons why certain results occur. We introduce a novel taxonomy to analyse replicating studies and discuss the importance of replicating and reproducing previous work. We describe what is the state of the art in collecting and sharing programming data. To better understand the challenges involved in replicating or reproducing existing studies, we report our experiences from three case studies using programming data. Finally, we present a discussion of future directions for the education and research community.

[1]  Nickolas J. G. Falkner,et al.  Increasing the effectiveness of automated assessment by increasing marking granularity and feedback units , 2014, SIGCSE.

[2]  Jürg Nievergelt,et al.  Kara, finite state machines, and the case for programming as part of general education , 2001, Proceedings IEEE Symposia on Human-Centric Computing Languages and Environments (Cat. No.01TH8587).

[3]  Ewan D. Tempero,et al.  Understanding the syntax barrier for novices , 2011, ITiCSE '11.

[4]  Neil Brown,et al.  Investigating novice programming mistakes: educator beliefs vs. student data , 2014, ICER '14.

[5]  Olly Gotel,et al.  Extending and contributing to an open source web-based system for the assessment of programming problems , 2007, PPPJ.

[6]  Daniel Zingaro,et al.  Facilitating code-writing in PI classes , 2013, SIGCSE '13.

[7]  Hendrik Blockeel,et al.  Experiment Databases , 2007, Inductive Databases and Constraint-Based Data Mining.

[8]  Carlos Fernandez-Medina,et al.  Assistance in computer programming learning using educational data mining and learning analytics , 2013, ITiCSE 2013.

[9]  Omar S. Gómez,et al.  Replication , Reproduction and Re-analysis : Three ways for verifying experimental , 2010 .

[10]  Stephen H. Edwards,et al.  Responses to adaptive feedback for software testing , 2014, ITiCSE '14.

[11]  Ma. Mercedes T. Rodrigo,et al.  Predicting at-risk novice Java programmers through the analysis of online protocols , 2011, ICER.

[12]  Allison Elliott Tew,et al.  A fresh look at novice programmers' performance and their teachers' expectations , 2013, ITiCSE -WGR '13.

[13]  Titus Winters,et al.  What do students know?: an outcomes-based assessment system , 2005, ICER '05.

[14]  Jaime Spacco,et al.  An exploration of error quotient in multiple contexts , 2015, Koli Calling.

[15]  Leonidas J. Guibas,et al.  Autonomously Generating Hints by Inferring Problem Solving Policies , 2015, L@S.

[16]  Peter Brusilovsky,et al.  Investigating Automated Student Modeling in a Java MOOC , 2014, EDM.

[17]  James B. Fenwick,et al.  ClockIt: collecting quantitative data on how beginning software developers really work , 2008, ITiCSE.

[18]  Lauri Malmi,et al.  On the use of resubmissions in automatic assessment systems , 2006, Comput. Sci. Educ..

[19]  Stephen H. Edwards,et al.  Increasing Adoption of Smart Learning Content for Computer Science Education , 2014, ITiCSE-WGR.

[20]  Stephen H. Edwards,et al.  Do student programmers all tend to write the same software tests? , 2014, ITiCSE '14.

[21]  Athanasios Tsintsifas,et al.  Automated assessment and experiences of teaching programming , 2005, JERC.

[22]  Peter J. Robinson.  MyPyTutor: an interactive tutorial system for Python , 2011, ACE 2011.

[23]  Stephen H. Edwards,et al.  Adaptively identifying non-terminating code when testing student programs , 2014, SIGCSE '14.

[24]  Susan Mengel,et al.  An Empirical Study of Iterative Improvement in Programming Assignments , 2015, SIGCSE.

[25]  Jaakko Kurhila,et al.  Using Students' Programming Behavior to Predict Success in an Introductory Mathematics Course , 2013, EDM.

[26]  Carlos Fernandez Medina,et al.  Assistance in computer programming learning using educational data mining and learning analytics , 2013, ITiCSE '13.

[27]  Ma. Mercedes T. Rodrigo,et al.  A detector for non-literal Java errors , 2010, Koli Calling.

[28]  Stephen H. Edwards,et al.  Comparing effective and ineffective behaviors of student programmers , 2009, ICER '09.

[29]  Matthew C. Jadud,et al.  An exploration of novice compilation behaviour in BlueJ , 2006 .

[30]  Stephen H. Edwards,et al.  Web-CAT: automatically grading programming assignments , 2008, ITiCSE.

[31]  Daniel Zingaro,et al.  Identifying challenging CS1 concepts in a large problem dataset , 2014, SIGCSE.

[32]  Pearl Brereton,et al.  Performing systematic literature reviews in software engineering , 2006, ICSE.

[33]  Patrick Jermann,et al.  Identifying Styles and Paths toward Success in MOOCs , 2015, EDM.

[34]  Ryan Shaun Joazeiro de Baker,et al.  Coarse-grained detection of student frustration in an introductory programming course , 2009, ICER '09.

[35]  Stephen H. Edwards,et al.  Effective and ineffective software testing behaviors by novice programmers , 2013, ICER '13.

[36]  Jussi Kasurinen,et al.  Estimating programming knowledge with Bayesian knowledge tracing , 2009, ITiCSE.

[37]  Petri Ihantola,et al.  How novices tackle their first lines of code in an IDE: analysis of programming session traces , 2014, Koli Calling.

[38]  James R. Curran,et al.  Data Mining for Generating Hints in a Python Tutor , 2010, EDM.

[39]  Lauri Malmi,et al.  Interaction and Feedback in Automatically Assessed Algorithm Simulation Exercises , 2003, J. Inf. Technol. Educ..

[40]  Ari Korhonen,et al.  How (not) to introduce badges to online exercises , 2014, SIGCSE.

[41]  Neil Brown,et al.  37 Million Compilations: Investigating Novice Programming Mistakes in Large-Scale Student Data , 2015, SIGCSE.

[42]  Jean Luca Bez,et al.  URI Online Judge Academic: A tool for algorithms and programming classes , 2014, 2014 9th International Conference on Computer Science & Education.

[43]  Juha Sorva,et al.  UUhistle: a software tool for visual program simulation , 2010, Koli Calling.

[44]  Ryan Shaun Joazeiro de Baker,et al.  Affective and behavioral predictors of novice programmer achievement , 2009, ITiCSE.

[45]  David Hovemeyer,et al.  CloudCoder: a web-based programming exercise system , 2013 .

[46]  Stephen H. Edwards,et al.  Exploring influences on student adherence to test-driven development , 2012, ITiCSE '12.

[47]  Yoshiaki Matsuzawa,et al.  Programming process visualizer: a proposal of the tool for students to observe their programming process , 2013, ITiCSE '13.

[48]  Bruce W. Weide,et al.  Which pointer errors do students make? , 2007, SIGCSE.

[49]  Matthew C. Jadud,et al.  Methods and tools for exploring novice compilation behaviour , 2006, ICER '06.

[50]  Valentina Dagiene,et al.  Observations from semi-automatic testing of program codes in the high school student maturity exam , 2010, Koli Calling.

[51]  Thomas Vikberg,et al.  Test My Code , 2013 .

[52]  Amruth N. Kumar.  Generation of problems, answers, grade, and feedback---case study of a fully automated tutor , 2005, JERC.

[53]  Lauri Malmi,et al.  TRAKLA2: a Framework for Automatically Assessed Visual Algorithm Simulation Exercises , 2003 .

[54]  Matthew C. Jadud.  A First Look at Novice Compilation Behaviour Using BlueJ , 2005, Comput. Sci. Educ..

[55]  James R. Curran,et al.  Data Mining for Individualised Hints in eLearning , 2009 .

[56]  Charles E. McDowell,et al.  Middle school students using Alice: what can we learn from logging data? , 2013, SIGCSE '13.

[57]  Gregory Dyke.  Which aspects of novice programmers' usage of an IDE predict learning outcomes , 2011, SIGCSE '11.

[58]  Amela Karahasanovic,et al.  Difficulties experienced by students in maintaining object-oriented systems: an empirical study , 2007 .

[59]  Gregorio Robles,et al.  Correlation between bug notifications, messages and participants in Debian's bug tracking system , 2007, ESEM 2007.

[60]  Stephen H. Edwards,et al.  Dereferee: exploring pointer mismanagement in student code , 2009, SIGCSE '09.

[61]  Georgios Evangelidis,et al.  A system for program visualization and problem-solving path assessment of novice programmers , 2001, ITiCSE '01.

[62]  Petri Ihantola,et al.  Recording and analyzing in-browser programming sessions , 2013, Koli Calling '13.

[63]  Andrew Luxton-Reilly,et al.  Enhancing syntax error messages appears ineffectual , 2014, ITiCSE '14.

[64]  Stephen H. Edwards,et al.  Impacts of adaptive feedback on teaching test-driven development , 2013, SIGCSE '13.

[65]  Charles E. McDowell,et al.  A First Step in Learning Analytics: Pre-processing Low-Level Alice Logging Data of Middle School Students , 2013, EDM 2013.

[66]  Andrew Lim,et al.  On automated grading of programming assignments in an academic institution , 2003, Comput. Educ..

[67]  Rachel Cardell-Oliver,et al.  How can software metrics help novice programmers , 2011, ACE 2011.

[68]  Peter Brusilovsky,et al.  Exploring Problem Solving Paths in a Java Programming Course , 2014, PPIG.

[69]  Lauri Malmi,et al.  How do students solve parsons programming problems?: an analysis of interaction traces , 2012, ICER '12.

[70]  Lauri Malmi,et al.  Beacon- and Schema-Based Method for Recognizing Algorithms from Students’ Source Code. , 2013, EDM 2013.

[71]  James B. Fenwick,et al.  Another look at the behaviors of novice programmers , 2009, SIGCSE '09.

[72]  Amela Karahasanovic,et al.  An Investigation into Keystroke Latency Metrics as an Indicator of Programming Performance , 2005, ACE.

[73]  Bertrand Meyer,et al.  An Incremental Hint System For Automated Programming Assignments , 2015, ITiCSE.

[74]  N. Black,et al.  The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. , 1998, Journal of epidemiology and community health.

[75]  Ewan D. Tempero,et al.  All syntax errors are not equal , 2012, ITiCSE '12.

[76]  Juha Sorva,et al.  Exploring programming misconceptions: an analysis of student mistakes in visual program simulation exercises , 2012, Koli Calling.

[77]  Davide Fossati,et al.  Towards improving programming habits to create better computer science course outcomes , 2013, ITiCSE '13.

[78]  Paul Roe,et al.  Program annotations: feedback for students learning to program , 2006 .

[79]  Stephen H. Edwards,et al.  Running students' software tests against each others' code: new life for an old "gimmick" , 2012, SIGCSE '12.

[80]  R. Nigel Horspool,et al.  Code Hunt: Experience with Coding Contests at Scale , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[81]  David Hovemeyer,et al.  Experiences with marmoset: designing and using an advanced submission and testing system for programming courses , 2006, ITICSE '06.

[82]  Per Runeson,et al.  Guidelines for conducting and reporting case study research in software engineering , 2009, Empirical Software Engineering.

[83]  Petri Ihantola,et al.  Analysis of source code snapshot granularity levels , 2014, SIGITE '14.

[84]  David Hovemeyer,et al.  Analyzing Student Work Patterns Using Programming Exercise Data , 2015, SIGCSE.

[85]  Lauri Malmi,et al.  Categorizing variations of student-implemented sorting algorithms , 2012, Comput. Sci. Educ..

[86]  Korbinian Riedhammer,et al.  Online identification of learner problem solving strategies using pattern recognition methods , 2010, ITiCSE '10.

[87]  C. Drummond.  Replicability is not Reproducibility:Nor is it Good Science , 2009 .

[88]  Stephen H. Edwards,et al.  A formative study of influences on student testing behaviors , 2014, SIGCSE.

[89]  Philip M. Johnson.  Requirement and Design Trade-offs in Hackystat: An In-Process Software Engineering Measurement and Analysis System , 2007, First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007).

[90]  David Hovemeyer,et al.  An open platform for managing short programming exercises , 2013, ICER '13.

[91]  Peter Hubwieser,et al.  Evaluation of Source Code with Item Response Theory , 2015, ITiCSE.

[92]  Ulrich Kiesmüller,et al.  Diagnosing Learners’ Problem-Solving Strategies Using Learning Environments with Algorithmic Problems in Secondary Education , 2009, TOCE.

[93]  Neil Brown,et al.  Blackbox: a large scale repository of novice programmers' activity , 2014, SIGCSE.

[94]  Petri Ihantola,et al.  A mobile learning application for parsons problems with automatic feedback , 2012, Koli Calling.

[95]  Ewan D. Tempero,et al.  CodeWrite: supporting student-driven practice of java , 2011, SIGCSE.

[96]  Raymond Lister.  CS EDUCATION RESEARCHThe naughties in CSEd research: a retrospective , 2010, INROADS.

[97]  Angela Carbone,et al.  Implementation of a smart lab for teachers of novice programmers , 2012, ACE 2012.

[98]  Lauri Malmi,et al.  Characterizing research in computing education: a preliminary analysis of the literature , 2010, ICER '10.

[99]  Frederick W. B. Li,et al.  No tests required: comparing traditional and dynamic predictors of programming success , 2014, SIGCSE.

[100]  Henrik Nygren,et al.  Identification of programmers from typing patterns , 2015, Koli Calling.

[101]  Prasun Dewan,et al.  Towards Incremental Separation of Surmountable and Insurmountable Programming Difficulties , 2015, SIGCSE.

[102]  Paulo Blikstein,et al.  Modeling how students learn to program , 2012, SIGCSE '12.

[103]  Tarek Hegazy,et al.  The CourseMarker CBA System: Improvements over Ceilidh , 2004, Education and Information Technologies.

[104]  Nickolas J. G. Falkner,et al.  A fast measure for identifying at-risk students in computer science , 2012, ICER '12.

[105]  Qin Zhang,et al.  Practical automated process and product metric collection and analysis in a classroom setting: lessons learned from Hackystat-UH , 2004, Proceedings. 2004 International Symposium on Empirical Software Engineering, 2004. ISESE '04..

[106]  Stephen H. Edwards,et al.  Mining Data from an Automated Grading and Testing System by Adding Rich Reporting Capabilities , 2008, EDM.

[107]  Irena Koprinska,et al.  Students at Risk: Detection and Remediation , 2015, EDM.

[108]  Michael C. Hughes,et al.  String formatting considered harmful for novice programmers , 2010, Comput. Sci. Educ..

[109]  Ewan D. Tempero,et al.  On the differences between correct student solutions , 2013, ITiCSE '13.

[110]  Michael C. Frank,et al.  Estimating the reproducibility of psychological science , 2015, Science.

[111]  Bertrand Meyer,et al.  SPOC-supported introduction to programming , 2014, ITiCSE '14.