Automated Assessment of Complex Programming Tasks Using SIETTE

This paper presents an innovative method to tackle the automatic evaluation of programming assignments with an approach based on well-founded assessment theories (Classical Test Theory (CTT) and Item Response Theory (IRT)) instead of heuristic assessment as in other systems. CTT and/or IRT are used to grade the results of different items of evidence obtained from students’ results. The methodology consists of considering program proofs as items, calibrating them, and obtaining the score using CTT and/or IRT procedures. These procedures measure overall validity reliability as well as diagnose the quality of each proof (item). The evidence is obtained through program proofs. The SIETTE system collects and processes all data to calculate the student knowledge level. This innovative method for programming task evaluation makes it possible to deploy the whole artillery developed in this research field over the last few decades. To the best of our knowledge, this is a new and original contribution in the area of programming assessment.

[1]  Shahida Sulaiman,et al.  Automatic programming assessment and test data generation a review on its approaches , 2010, 2010 International Symposium on Information Technology.

[2]  Moffat Mathews,et al.  Data calibration for statistical-based assessment in constraint-based tutors , 2016, Knowl. Based Syst..

[3]  Stellan Ohlsson,et al.  Constraint-Based Student Modeling , 1994 .

[4]  Julio C. Caiza,et al.  PROGRAMMING ASSIGNMENTS AUTOMATIC GRADING: REVIEW OF TOOLS AND IMPLEMENTATIONS , 2013 .

[5]  Vreda Pieterse,et al.  Automated Assessment of Programming Assignments , 2013, CSERC.

[6]  Enric Rodríguez-Carbonell,et al.  Jutge.org: Characteristics and Experiences , 2018, IEEE Transactions on Learning Technologies.

[7]  Eduardo Guzmán,et al.  Exploring Quality of Constraints for Assessment in Problem Solving Environments , 2012, ITS.

[8]  Robert J. Mislevy,et al.  Evidence-Centered Assessment Design , 2017 .

[9]  Michael J. Zieky.  An introduction to the use of evidence-centered design in test development , 2014 .

[10]  José Paulo Leal,et al.  Mooshak: a Web‐based multi‐site programming contest system , 2003, Softw. Pract. Exp..

[11]  Jéan H. Greyling,et al.  Marking student programs using graph similarity , 2010, Comput. Educ..

[12]  Pieter Abbeel,et al.  Gradescope: A Fast, Flexible, and Fair System for Scalable Assessment of Handwritten Work , 2017, L@S.

[13]  Viggo Kann,et al.  Five years with kattis — Using an automated assessment system in teaching , 2011, 2011 Frontiers in Education Conference (FIE).

[14]  R. Almond,et al.  Focus Article: On the Structure of Educational Assessments , 2003 .

[15]  Ricardo Conejo,et al.  SIETTE: A Web-Based Tool for Adaptive Testing , 2004, Int. J. Artif. Intell. Educ..

[16]  Nathan Griffiths,et al.  The boss online submission and assessment system , 2005, JERC.

[17]  Norazah Yusof,et al.  Java Programming Assessment Tool for Assignment Module in Moodle E-learning System , 2012 .

[18]  Tapio Salakoski,et al.  Interactive Learning Content for Introductory Computer Science Course Using the ViLLE Exercise Framework , 2015, 2015 International Conference on Learning and Teaching in Computing and Engineering.

[19]  Christopher Douce,et al.  Automatic test-based assessment of programming: A review , 2005, JERC.

[20]  Petri Ihantola,et al.  Review of recent systems for automatic assessment of programming assignments , 2010, Koli Calling.

[21]  Xiaohong Su,et al.  Semantic similarity-based grading of student programs , 2007, Inf. Softw. Technol..

[22]  Eduardo Guzmán,et al.  The SIETTE Automatic Assessment Environment , 2015, International Journal of Artificial Intelligence in Education.

[23]  Eduardo Guzmán,et al.  Student Knowledge Diagnosis Using Item Response Theory and Constraint-Based Modeling , 2009, AIED.

[24]  Seiji Isotani,et al.  Teaching novice programmers using ProgTest , 2015, Int. J. Knowl. Learn..

[25]  Eladio Gutiérrez,et al.  A new Moodle module supporting automatic verification of VHDL-based assignments , 2010, Comput. Educ..

[26]  Viktor Kuncak,et al.  Software verification and graph similarity for automated evaluation of students' assignments , 2012, Inf. Softw. Technol..