Grading the Graders: Motivating Peer Graders in a MOOC

In this paper, we detail our efforts at creating and running a controlled study designed to examine how students in a MOOC might be motivated to do a better job during peer grading. This study involves more than one thousand students of a popular MOOC. We ask two specific questions: (1) When a student knows that his or her own peer grading efforts are being examined by peers, does this knowledge alone tend to motivate the student to do a better job when grading assignments? And (2) when a student not only knows that his or her own peer grading efforts are being examined by peers, but he or she is also given a number of other peer grading efforts to evaluate (so the peer graders see how other peer graders evaluate assignments), do both of these together tend to motivate the student to do a better job when grading assignments? We find strong statistical evidence that ``grading the graders'' does in fact tend to increase the quality of peer grading.

[1]  B. Efron,et al.  The Jackknife: The Bootstrap and Other Resampling Plans. , 1983 .

[2]  Sidney Addelman,et al.  trans-Dimethanolbis(1,1,1-trifluoro-5,5-dimethylhexane-2,4-dionato)zinc(II) , 2008, Acta crystallographica. Section E, Structure reports online.

[3]  Sarah Warshauer Freedman,et al.  Peer Response Groups in the Writing Classroom: Theoretic Foundations and New Directions , 1988 .

[4]  K. Hew,et al.  Students’ and instructors’ use of massive open online courses (MOOCs): Motivations and challenges , 2014 .

[5]  N. Falchikov.  Peer Feedback Marking: Developing Peer Assessment , 1995 .

[6]  K. Topping.  Peer Assessment Between Students in Colleges and Universities , 1998 .

[7]  Jure Leskovec,et al.  Steering user behavior with badges , 2013, WWW.

[8]  Justin Cheng,et al.  Peer and self assessment in massive online classes , 2013, ACM Trans. Comput. Hum. Interact..

[9]  David E. Pritchard,et al.  Studying Learning in the Worldwide Classroom Research into edX's First MOOC. , 2013 .

[10]  Thorsten Dickhaus,et al.  Simultaneous Statistical Inference , 2014, Springer Berlin Heidelberg.

[11]  T. Risley,et al.  Peer tutoring among elementary students: educational benefits to the tutor. , 1977, Journal of applied behavior analysis.

[12]  Chris Piech,et al.  Deconstructing disengagement: analyzing learner subpopulations in massive open online courses , 2013, LAK '13.

[13]  Dominique Sluijsmans,et al.  Creating a Learning Environment by Using Self-, Peer- and Co-Assessment , 1998 .

[14]  James A. Keaten,et al.  A Field Investigation of Peer Assessment as Part of the Student Group Grading Process. , 1993 .

[15]  Mitchell J. Nathan,et al.  Improving Students’ Learning With Effective Learning Techniques: Promising Directions From Cognitive and Educational Psychology , 2012 .

[16]  Yi-hsiu Lai,et al.  Which do students prefer to evaluate their essays: Peers or computer program , 2010, Br. J. Educ. Technol..

[17]  Dympna Casey,et al.  Use of peer assessment as a student engagement strategy in nurse education. , 2011, Nursing & health sciences.

[18]  Scott Rixner,et al.  An environment for learning interactive programming , 2014, SIGCSE '14.

[19]  Rupert G. Miller.  Simultaneous Statistical Inference , 1966 .

[20]  N. Falchikov,et al.  Student Peer Assessment in Higher Education: A Meta-Analysis Comparing Peer and Teacher Marks , 2000 .

[21]  Jo Davies,et al.  Performance in e-learning: online participation and student grades , 2005, Br. J. Educ. Technol..

[22]  D. Sluijsmans,et al.  The use of self-, peer and co-assessment in higher education: A review , 1999 .

[23]  Gary L. May,et al.  Collaborative Peer Evaluation: Best Practices for Group Member Assessments , 2002 .

[24]  B. Efron.  The jackknife, the bootstrap, and other resampling plans , 1987 .

[25]  Arpita Ghosh,et al.  Incentives, gamification, and game theory: an economic approach to badge design , 2013, EC '13.

[26]  Peggy A. Ertmer,et al.  Using Peer Feedback to Enhance the Quality of Student Online Postings: An Exploratory Study , 2007, J. Comput. Mediat. Commun..

[27]  Jure Leskovec,et al.  Engaging with massive online courses , 2014, WWW.

[28]  Hugh Somervell,et al.  Issues in Assessment, Enterprise and Higher Education: the case for self‐peer and collaborative assessment , 1993 .