Building a Comprehensive Automated Programming Assessment System

Automated Programming Assessment Systems (APAS) are used for overcoming problems associated with manually managed programming assignments, such as objective and efficient assessing in large classes and providing timely and helpful feedback. In this paper we survey the literature and software in this field and identify the set of necessary features that make APAS comprehensive – such that it can support all key stages in the assessment process. Put differently, comprehensive APAS is generic enough to meet the demands of “any” computer science course. Despite the vast number of publications, the choice of software turns out to be very limited. We contribute by developing Edgar, a comprehensive open-source APAS which, to the best of our knowledge, exceeds any other similar free and/or open-source tool. Edgar is the result of three years of development and usage in, for the time being, eight courses dealing with various programming languages and paradigms (C, Java, SQL, etc.). Edgar supports various text-based programming languages, multi-correct multiple-choice questions, provides rich exam logging and monitoring infrastructure to prevent potential fraudulent behaviour, and subsequent data analysis and visualization of students’ scores, exams, question quality, etc. It can be deployed on all major operating systems and is written in a modular fashion so that it can be adjusted and scaled to a custom fit. We comment on the architecture and present data from real-world use-cases to support these claims. Edgar is in active use today (1000+ students per semester) and it is being constantly developed with new features.

[1]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[2]  Tarek Hegazy,et al.  The CourseMarker CBA System: Improvements over Ceilidh , 2004, Education and Information Technologies.

[3]  John Hughes,et al.  Security Assertion Markup Language (SAML) 2.0 Technical Overview , 2004 .

[4]  Manfred Widera,et al.  Partial Specifications of Program Properties , 2004 .

[5]  Georg Rasch,et al.  Probabilistic Models for Some Intelligence and Attainment Tests , 1981, The SAGE Encyclopedia of Research Design.

[6]  Angelo Kyrilov,et al.  Do students need detailed feedback on programming exercises and can automated assessment systems provide it , 2016 .

[7]  Athanasios Tsintsifas,et al.  Authoring diagram-based CBA with CourseMarker , 2009, Comput. Educ..

[8]  Jack Hollingsworth,et al.  Automatic graders for programming classes , 1960, Commun. ACM.

[9]  Steve Benford,et al.  The Ceilidh system for the automatic grading of students on programming courses , 1995, ACM-SE 33.

[10]  Michael Striewe,et al.  An architecture for modular grading and feedback generation for complex exercises , 2016, Sci. Comput. Program..

[11]  José Luis Fernández Alemán.  Automated Assessment in a Programming Tools Course , 2011, IEEE Trans. Educ..

[12]  Varmo Vene,et al.  Comparing different styles of automated feedback for programming exercises , 2017, Koli Calling.

[13]  Josep Silva,et al.  Automatic assessment of Java code , 2018, Comput. Lang. Syst. Struct..

[14]  Angelo Kyrilov,et al.  Binary instant feedback on programming exercises can reduce student engagement and promote cheating , 2015, Koli Calling.

[15]  Josep Silva,et al.  Computer assisted self-assessment of programming code: A report on university students experience and opinion , 2016, 2016 15th International Conference on Information Technology Based Higher Education and Training (ITHET).

[16]  Ivan Pribela,et al.  A Temporal File System for Student's Assignments in The System Svetovid , 2017, SQAMIA.

[17]  Anthony M. Sloane,et al.  A Web-based System for Automatic Program Critiquing , 2003, ACE.

[18]  Nikolai Tillmann,et al.  Measuring Code Behavioral Similarity for Programming and Software Engineering Education , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C).

[19]  John English,et al.  Experiences of Using Automated Assessment in Computer Science Courses , 2015 .

[20]  Johan Jeuring,et al.  A Systematic Literature Review of Automated Feedback Generation for Programming Exercises , 2018, ACM Trans. Comput. Educ..

[21]  Stephen H. Edwards,et al.  Work-in-progress: program grading and feedback generation with Web-CAT , 2014, L@S.

[22]  Ibrahim Albluwi.  A Closer Look at the Differences Between Graders in Introductory Computer Science Exams , 2018, IEEE Transactions on Education.

[23]  Lauri Malmi,et al.  On the use of resubmissions in automatic assessment systems , 2006, Comput. Sci. Educ..

[24]  Ellen Francine Barbosa,et al.  A Systematic Literature Review of Assessment Tools for Programming Assignments , 2016, 2016 IEEE 29th International Conference on Software Engineering Education and Training (CSEET).

[25]  Lauri Malmi,et al.  Enhancing learning management systems to better support computer science education , 2008, ACM SIGCSE Bull..

[26]  Xue Bai,et al.  ENHANCING THE LEARNING PROCESS IN PROGRAMMING COURSES THROUGH AN AUTOMATED FEEDBACK AND ASSIGNMENT MANAGEMENT SYSTEM , 2016 .

[27]  Robert E. Noonan.  The back end of a grading system , 2006, SIGCSE '06.

[28]  Dietmar F. Rösner,et al.  E-Assessment as a Service , 2011, IEEE Transactions on Learning Technologies.

[29]  Michael Luck,et al.  Automatic submission in an evolutionary approach to computer science teaching computers and educatio , 1995 .

[30]  Qiang Hao,et al.  Investigating the Essential of Meaningful Automated Formative Feedback for Programming Assignments , 2019, 2019 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[31]  Ljiljana Brkic,et al.  Setting Up Automated Programming Assessment System for Higher Education Database Course , 2017 .

[32]  José Paulo Leal,et al.  Mooshak: a Web‐based multi‐site programming contest system , 2003, Softw. Pract. Exp..

[33]  Christopher Douce,et al.  Automatic test-based assessment of programming: A review , 2005, JERC.

[34]  Petri Ihantola,et al.  Review of recent systems for automatic assessment of programming assignments , 2010, Koli Calling.

[35]  Roozbeh Matloobi,et al.  Extensions to Generic Automated Marking Environment: GAME-2+ , 2009 .

[36]  Vreda Pieterse,et al.  Automated Assessment of Programming Assignments , 2013, CSERC.

[37]  P. Pale,et al.  AuResS: The audience response system , 2012, Proceedings ELMAR-2012.

[38]  Nathan Griffiths,et al.  The boss online submission and assessment system , 2005, JERC.

[39]  Xavier Burgués Illa,et al.  A software tool for e-assessment of relational database skills , 2016 .

[40]  Evandro Costa,et al.  An Adaptive Approach to Provide Feedback for Students in Programming Problem Solving , 2019, ITS.

[41]  Bernard Blackham.  A New Contest Sandbox , 2012 .

[42]  Muath Alkhalaf,et al.  Automated Grading Systems for Programming Assignments: A Literature Review , 2019, International Journal of Advanced Computer Science and Applications.

[43]  Julio C. Caiza,et al.  PROGRAMMING ASSIGNMENTS AUTOMATIC GRADING: REVIEW OF TOOLS AND IMPLEMENTATIONS , 2013 .

[44]  R. Lobb,et al.  Coderunner , 2016, Inroads.

[45]  Kirsti Ala-Mutka,et al.  A Survey of Automated Assessment Approaches for Programming Assignments , 2005, Comput. Sci. Educ..

[46]  Janet Liebenberg,et al.  Investigating the Feasibility of Automatic Assessment of Programming Tasks , 2018, J. Inf. Technol. Educ. Innov. Pract..

[47]  Nickolas J. G. Falkner,et al.  Increasing the effectiveness of automated assessment by increasing marking granularity and feedback units , 2014, SIGCSE.

[48]  Nigel Stanger,et al.  Semi-automated assessment of SQL schemas via database unit testing , 2018 .

[49]  Tapio Auvinen,et al.  Harmful Study Habits in Online Learning Environments with Automatic Assessment , 2015, 2015 International Conference on Learning and Teaching in Computing and Engineering.

[50]  Károly Nehéz,et al.  Grading Java code submissions in MeMOOC , 2018 .

[51]  David Hovemeyer,et al.  Experiences with marmoset: designing and using an advanced submission and testing system for programming courses , 2006, ITICSE '06.