Probabilistic Graphical Models for Boosting Cardinal and Ordinal Peer Grading in MOOCs

With the enormous scale of massive open online courses (MOOCs), peer grading is vital for addressing the assessment challenge for open-ended assignments or exams while at the same time providing students with an effective learning experience through involvement in the grading process. Most existing MOOC platforms use simple schemes for aggregating peer grades, e.g., taking the median or mean. To enhance these schemes, some recent research attempts have developed machine learning methods under either the cardinal setting (for absolute judgment) or the ordinal setting (for relative judgment). In this paper, we seek to study both cardinal and ordinal aspects of peer grading within a common framework. First, we propose novel extensions to some existing probabilistic graphical models for cardinal peer grading. Not only do these extensions give superior performance in cardinal evaluation, but they also outperform conventional ordinal models in ordinal evaluation. Next, we combine cardinal and ordinal models by augmenting ordinal models with cardinal predictions as prior. Such combination can achieve further performance boosts in both cardinal and ordinal evaluations, suggesting a new research direction to pursue for peer grading on MOOCs. Extensive experiments have been conducted using real peer grading data from a course called "Science, Technology, and Society in China I" offered by HKUST on the Coursera platform.

[1]  R. A. Bradley,et al.  RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS , 1952 .

[2]  Stevan Harnad,et al.  The invisible hand of peer review , 1998 .

[3]  Fytton Rowland,et al.  The peer‐review process , 2002, Learn. Publ..

[4]  Nardine Osman,et al.  Collaborative Assessment , 2014, EDM.

[5]  P. Sadler,et al.  The Impact of Self- and Peer-Grading on Student Learning , 2006 .

[6]  Martin J. Wainwright,et al.  When is it Better to Compare than to Score? , 2014, ArXiv.

[7]  William Barnett,et al.  The modern theory of consumer behavior: Ordinal or cardinal? , 2003 .

[8]  Thorsten Joachims,et al.  Optimizing search engines using clickthrough data , 2002, KDD.

[9]  Manolis Mavrikis,et al.  Proceedings of the 7th International Conference on Educational Data Mining. , 2014, EDM 2014.

[10]  David Maxwell Chickering,et al.  Here or There , 2008, ECIR.

[11]  Wei Chu,et al.  Preference learning with Gaussian processes , 2005, ICML.

[12]  Kannan Ramchandran,et al.  A Case for Ordinal Peer-evaluation in MOOCs , 2013 .

[13]  Peter J. Coughlin,et al.  Probabilistic Voting Theory , 1992 .

[14]  Rajeev Motwani,et al.  The PageRank Citation Ranking : Bringing Order to the Web , 1999, WWW 1999.

[15]  L. A. Goodman,et al.  Social Choice and Individual Values , 1951 .

[16]  Michael Vitale,et al.  The Wisdom of Crowds , 2015, Cell.

[17]  Peter J. Coughlin,et al.  Probabilistic Voting Theory. , 1993 .

[18]  Javed A. Aslam,et al.  Models for metasearch , 2001, SIGIR '01.

[19]  Eyke Hüllermeier,et al.  Preference Learning , 2005, Künstliche Intell..

[20]  Eyke Hllermeier,et al.  Preference Learning , 2010 .

[21]  Luca de Alfaro,et al.  CrowdGrader: a tool for crowdsourcing the evaluation of homework assignments , 2014, SIGCSE.

[22]  Toby Walsh,et al.  The PeerRank Method for Peer Assessment , 2014, ECAI.

[23]  K. Arrow,et al.  Social Choice and Individual Values , 1951 .

[24]  Jure Leskovec,et al.  Engaging with massive online courses , 2014, WWW.

[25]  Steve Joordens,et al.  Peering into large lectures: examining peer and expert mark agreement using peerScholar, an online peer assessment tool , 2008, J. Comput. Assist. Learn..

[26]  David Maxwell Chickering,et al.  Here or there: preference judgments for relevance , 2008 .

[27]  Moni Naor,et al.  Rank aggregation methods for the Web , 2001, WWW '01.

[28]  R. A. Bradley,et al.  Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons , 1952 .

[29]  T. Jefferson,et al.  Peer Review in Health Sciences , 1999 .

[30]  Zhenghao Chen,et al.  Tuned Models of Peer Assessment in MOOCs , 2013, EDM.

[31]  Gordon D. A. Brown,et al.  Absolute identification by relative judgment. , 2005, Psychological review.

[32]  S. Ceci,et al.  Peer-review practices of psychological journals: The fate of published articles, submitted again , 1982, Behavioral and Brain Sciences.

[33]  Frank Wilcoxon,et al.  Rank Correlation Method , 1955 .

[34]  Luca de Alfaro,et al.  CrowdGrader: Crowdsourcing the Evaluation of Homework Assignments , 2013, ArXiv.

[35]  Thorsten Joachims,et al.  Methods for ordinal peer grading , 2014, KDD.

[36]  Eyke Hüllermeier,et al.  Pairwise Preference Learning and Ranking , 2003, ECML.