Automated Assessment of Computer Programming Practices: The 8-Years UNED Experience

The increasing popularity of distance education poses exciting new challenges. In particular, current pedagogical paradigms, such as competency-based education, require students’ continuous evaluation. That is, to master skills, students need to receive constant feedback to guide their experimentation processes. However, teaching teams are usually under-dimensioned to support the large number of students that online courses usually have. This paper presents the approach we have adopted at the National University of Distance Education to overcome this problem for the case of computer programming practices, which complements human evaluation with an automatic assessment system. The paper describes our system and reports its benefits with an empirical study from 2011 to 2018 that involved 14,944 students.

[1]  Marco Aurélio Gerosa,et al.  Visual programming and automatic evaluation of exercises: An experience with a STEM course , 2016, 2016 IEEE Frontiers in Education Conference (FIE).

[2]  Darlinda Moreira.  From On-Campus to Online: A Trajectory of Innovation, Internationalization and Inclusion , 2016 .

[3]  M.A. Vouk,et al.  On effectiveness of pairwise methodology for testing network-centric software , 2005, 2005 International Conference on Information and Communication Technology.

[4]  Yu Lei,et al.  Practical Combinatorial Testing , 2010 .

[5]  Rafi Nachmias,et al.  Effectiveness of learning in online academic courses compared with face-to-face courses in higher education , 2018, J. Comput. Assist. Learn..

[6]  Dominique Thiébaut.  Automatic evaluation of computer programs using Moodle's virtual programming lab (VPL) plug-in , 2015 .

[7]  David Jackson,et al.  Grading student programs using ASSYST , 1997, SIGCSE '97.

[8]  Saša Mladenović,et al.  Undergraduate programming courses, students' perception and success , 2015 .

[9]  Rachid Benabbou,et al.  Automatic program assessment using static and dynamic analysis , 2015, 2015 Third World Conference on Complex Systems (WCCS).

[10]  A. Jefferson Offutt,et al.  Combination testing strategies: a survey , 2005, Softw. Test. Verification Reliab..

[11]  Andreas Seitz,et al.  ArTEMiS: An Automatic Assessment Management System for Interactive Learning , 2018, SIGCSE.

[12]  Victor C. S. Lee,et al.  ViDA: A virtual debugging advisor for supporting learning in computer programming courses , 2018, J. Comput. Assist. Learn..

[13]  Josep Silva,et al.  Automatic assessment of Java code , 2018, Comput. Lang. Syst. Struct..

[14]  D. Richard Kuhn,et al.  Software fault interactions and implications for software testing , 2004, IEEE Transactions on Software Engineering.

[15]  Daniel Lehmann,et al.  Feedback-directed differential testing of interactive debuggers , 2018, ESEC/SIGSOFT FSE.

[16]  Fabio A. González,et al.  Continuous assessment in a computer programming course supported by a software tool , 2018, Comput. Appl. Eng. Educ..

[17]  Alda Lopes Gançarski,et al.  A Flexible Dynamic System for Automatic Grading of Programming Exercises , 2013, SLATE.

[18]  Cedric E. Ginestet.  ggplot2: Elegant Graphics for Data Analysis , 2011 .

[19]  Niklaus Wirth,et al.  Program development by stepwise refinement , 1971, CACM.

[20]  S. Weisberg,et al.  Companion to Applied Regression , 2015 .

[21]  Gregorio Robles,et al.  Dr. Scratch: Automatic Analysis of Scratch Projects to Assess and Foster Computational Thinking , 2015 .

[22]  Muhammad Ali Gulzar,et al.  Perception and Practices of Differential Testing , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP).

[23]  Johan Jeuring,et al.  Towards a Systematic Review of Automated Feedback Generation for Programming Exercises , 2016, ITiCSE.

[24]  Ying Liu,et al.  Assessment of programming language learning based on peer code review model: Implementation and experience report , 2012, Comput. Educ..

[25]  Michael Striewe,et al.  A Review of Static Analysis Approaches for Programming Exercises , 2014, CAA.

[26]  William Doherty,et al.  An analysis of multiple factors affecting retention in Web-based community college courses , 2006, Internet High. Educ..

[27]  Patrick Jermann,et al.  A Comparison between Two Automatic Assessment Approaches for Programming: An Empirical Study on MOOCs , 2018, J. Educ. Technol. Soc..

[28]  Nur'Aini Abdul Rashid,et al.  A Framework of an Automatic Assessment System for Learning Programming , 2016 .

[29]  Kirsti Ala-Mutka,et al.  A Survey of Automated Assessment Approaches for Programming Assignments , 2005, Comput. Sci. Educ..

[30]  Cem Kaner,et al.  The Domain Testing Workbook , 2013 .

[31]  Christopher Douce,et al.  Automatic test-based assessment of programming: A review , 2005, JERC.

[32]  Petri Ihantola,et al.  Review of recent systems for automatic assessment of programming assignments , 2010, Koli Calling.

[33]  Yu Lei,et al.  Introduction to Combinatorial Testing , 2013 .

[34]  Zhen Chen,et al.  Automatic assessment of OpenGL computer graphics assignments , 2018, ITiCSE.

[35]  Ellen Francine Barbosa,et al.  A Systematic Literature Review of Assessment Tools for Programming Assignments , 2016, 2016 IEEE 29th International Conference on Software Engineering Education and Training (CSEET).

[36]  Arnab Bhattacharya,et al.  Automatic Grading and Feedback using Program Repair for Introductory Programming Courses , 2017, ITiCSE.

[37]  Nathalia da Cruz Alves,et al.  CodeMaster - Automatic Assessment and Grading of App Inventor and Snap! Programs , 2018, Informatics Educ..

[38]  Christian Kästner.  Differential Testing for Variational Analyses: Experience from Developing KConfigReader , 2017, ArXiv.

[39]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[40]  Varun Aggarwal,et al.  Question Independent Grading using Machine Learning: The Case of Computer Program Grading , 2016, KDD.

[41]  Shuchi Grover,et al.  Measuring Student Learning in Introductory Block-Based Programming: Examining Misconceptions of Loops, Variables, and Boolean Logic , 2017, SIGCSE.

[42]  Marco Temperini,et al.  A Gamified Approach to Automated Assessment of Programming Assignments , 2018 .

[43]  Seiji Isotani,et al.  Teaching novice programmers using ProgTest , 2015, Int. J. Knowl. Learn..

[44]  Ushio Inoue,et al.  An Online Automated Scoring System for Java Programming Assignments , 2016 .

[45]  Jack Hollingsworth,et al.  Automatic graders for programming classes , 1960, Commun. ACM.

[46]  Frank C. Gomez,et al.  Rethinking Education in the Age of Technology: The Digital Revolution and Schooling in America , 2016 .

[47]  D. Richard Kuhn,et al.  Pseudo-Exhaustive Testing for Software , 2006, 2006 30th Annual IEEE/NASA Software Engineering Workshop.