A Protocol for Simulated Experimentation of Automated Grading Systems

Educational systems providing Automated Grading can be very useful for both learner and teacher. After their design and implementation, systems providing automated grading have to be tested and validated, which let the need for real world experiments arise. In this paper we describe an approach to “simulated experiments” that we hope can ease the task of the researchers developing the above mentioned systems. The concept of our proposal is in defining a simulated class by means of statistical distributions of the features that model the student and her/his work in the system. Since such features can be very different from a system to another, we try and keep the description of our framework as general as possible, and define a simulated class by the distribution of the grades that the learners should get (and the system should infer, if correctly working). We show the use of our approach with two systems, that do grading and model the students quite differently. We conclude that our simulation framework can be beneficial in validating an automated grading system, and in allowing to reflect on possible updates of the system, to make its grading more correct.

[1]  Ig Ibert Bittencourt,et al.  Does peer assessment in on-line learning environments work? A systematic review of the literature , 2016, Computers in Human Behavior.

[2]  Jeffrey S. Kane,et al.  Methods of peer assessment. , 1978 .

[3]  Pierpaolo Vittorini,et al.  The Automated Grading of R Code Snippets: Preliminary Results in a Course of Health Informatics , 2019, MIS4TEL.

[4]  R. Nelsen An Introduction to Copulas , 1998 .

[5]  H. Akaike A new look at the statistical model identification , 1974 .

[6]  B. Silverman Density estimation for statistics and data analysis , 1986 .

[7]  M. Crowder Maximum Likelihood Estimation for Dependent Observations , 1976 .

[8]  Vallipuram Muthukkumarasamy,et al.  Performance analysis of GAME: A generic automated marking environment , 2008, Comput. Educ..

[9]  Maria De Marsico,et al.  Supporting Mediated Peer-Evaluation to Grade Answers to Open-Ended Questions , 2017 .

[10]  Zhenghao Chen,et al.  Tuned Models of Peer Assessment in MOOCs , 2013, EDM.

[11]  Cary Moskovitz,et al.  Calibrated Peer Review Assignments in Science Courses: Are They Designed to Promote Critical Thinking and Writing Skills? , 2008 .

[12]  Hoi K. Suen,et al.  Peer assessment for massive open online courses (MOOCs) , 2014 .

[13]  M. Delignette-Muller,et al.  fitdistrplus: An R Package for Fitting Distributions , 2015 .

[14]  G. Schwarz Estimating the Dimension of a Model , 1978 .

[15]  J. Metcalfe,et al.  Metacognition : knowing about knowing , 1994 .

[16]  Richard J. Rossi,et al.  Mathematical Statistics , 2018 .

[17]  Benjamin S. Bloom,et al.  A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives , 2000 .

[18]  P. Sadler,et al.  The Impact of Self- and Peer-Grading on Student Learning , 2006 .

[19]  Marco Temperini,et al.  Peer-assessment and grading of open answers in a web-based e-learning setting , 2013, 2013 12th International Conference on Information Technology Based Higher Education and Training (ITHET).

[20]  Rodney C. Wolff,et al.  Methods for estimating a conditional distribution function , 1999 .

[21]  Kirsti Ala-Mutka,et al.  A Survey of Automated Assessment Approaches for Programming Assignments , 2005, Comput. Sci. Educ..

[22]  Lan Li,et al.  Assessor or assessee: How student learning improves by giving and receiving peer feedback , 2010, Br. J. Educ. Technol..

[23]  David R. Anderson,et al.  Model selection and multimodel inference : a practical information-theoretic approach , 2003 .