Incentivizing Peer Grading in MOOCS: An Audit Game Approach

In Massively Open Online Courses (MOOCs) TA resources are limited; most MOOCs use peer assessments to grade assignments. Students have to divide up their time between working on their own homework and grading others. If there is no risk of being caught and penalized, students have no reason to spend any time grading others. Course staff want to incentivize students to balance their time between course work and peer grading. They may do so by auditing students, ensuring that they perform grading correctly. One would not want students to invest too much time on peer grading, as this would result in poor course performance. We present the first model of strategic auditing in peer grading, modeling the student's choice of effort in response to a grader's audit levels as a Stackelberg game with multiple followers. We demonstrate that computing the equilibrium for this game is computationally hard. We then provide a PTAS in order to compute an approximate solution to the problem of allocating audit levels. However, we show that this allocation does not necessarily maximize social welfare; in fact, there exist settings where course auditor utility is arbitrarily far from optimal under an approximately optimal allocation. To circumvent this issue, we present a natural condition that guarantees that approximately optimal TA allocations guarantee approximately optimal welfare for the course auditors.

[1]  Nicolas Christin,et al.  Audit Games with Multiple Defender Resources , 2014, AAAI.

[2]  Kevin Leyton-Brown,et al.  Mechanical TA: Partially Automated High-Stakes Peer Grading , 2015, SIGCSE.

[3]  Bo An,et al.  Refinement of Strong Stackelberg Equilibria in Security Games , 2011, AAAI.

[4]  Michael I. Jordan,et al.  Advances in Neural Information Processing Systems 30 , 1995 .

[5]  David S. Johnson,et al.  Computers and Intractability: A Guide to the Theory of NP-Completeness , 1978 .

[6]  Nicolas Christin,et al.  Audit Games , 2013, IJCAI.

[7]  Vincent Conitzer,et al.  Security Games with Multiple Attacker Resources , 2011, IJCAI.

[8]  Debmalya Mandal,et al.  A Stackelberg Game Approach for Incentivizing Participation in Online Educational Forums with Heterogeneous Student Population , 2015, AAAI.

[9]  Zhenghao Chen,et al.  Tuned Models of Peer Assessment in MOOCs , 2013, EDM.

[10]  Vincent Conitzer,et al.  Game-Theoretic Question Selection for Tests , 2013, IJCAI.

[11]  Manish Jain,et al.  Software Assistants for Randomized Patrol Planning for the LAX Airport Police and the Federal Air Marshal Service , 2010, Interfaces.

[12]  Christopher M. Bishop,et al.  Students, Teachers, Exams and MOOCs: Predicting and Optimizing Attainment in Web-Based Education Using a Probabilistic Graphical Model , 2014, ECML/PKDD.

[13]  Manish Jain,et al.  Computing optimal randomized resource allocations for massive security games , 2009, AAMAS 2009.

[14]  Manish Jain,et al.  Computing optimal randomized resource allocations for massive security games , 2009, AAMAS.

[15]  Milind Tambe.  Security and Game Theory: EFFICIENT ALGORITHMS FOR MASSIVE SECURITY GAMES , 2011 .

[16]  Boi Faltings,et al.  An incentive compatible reputation mechanism , 2003, AAMAS '03.

[17]  Arpita Ghosh,et al.  Incentivizing participation in online forums for education , 2013, EC '13.

[18]  Ariel D. Procaccia,et al.  Learning Optimal Commitment to Overcome Insecurity , 2014, NIPS.

[19]  Milind Tambe,et al.  Security and Game Theory - Algorithms, Deployed Systems, Lessons Learned , 2011 .

[20]  Milind Tambe,et al.  Stop the compartmentalization: unified robust algorithms for handling uncertainties in security games , 2014, AAMAS.