Is peer review an appropriate form of assessment in a MOOC? Student participation and performance in formative peer review

Abstract Many aspects of higher education must be reconceptualised for massive open online courses (MOOCs). Formative and summative assessment of qualitative work in particular requires novel approaches to cope with the numbers involved. Peer review has been proposed as one solution, and has been widely adopted by major MOOC providers, but there is currently little evidence about whether it is appropriate or under what conditions. Here, we examine student participation, performance and opinions of a peer review task in a biomedical science MOOC. We evaluate data from approximately 200 student topic summaries and 300 qualitative peer reviews of those summaries, and compare these to student demographic data (gender, age, employment status, education, national language) and to performance in multiple choice tests. We show that higher performance in the written topic summary correlated with both higher participation in the peer review task and with writing higher quality peer reviews. Qualitative analysis of student comments revealed that student opinion on the usefulness of the peer review task was mixed: some strongly believed it benefitted their learning, while others did not find it useful or did not participate. We suggest instructional design strategies to improve student participation and increase learning gain from peer review in the MOOC context.

[1]  John A. Dutton.  Peer Grading in a MOOC : Reliability , Validity , and Perceived Effects , 2014 .

[2]  Wilfried Admiraal,et al.  Assessment in Massive Open Online Courses , 2015 .

[3]  Gayle S. Christensen,et al.  The MOOC Phenomenon: Who Takes Massive Open Online Courses and Why? , 2013 .

[4]  J. Morris,et al.  Peer assessment: a missing link between teaching and learning? A review of the literature. , 2001, Nurse education today.

[5]  Ebba Ossiannilsson,et al.  Analysis of MOOCs practices from the perspective of learner experiences and quality culture , 2015 .

[6]  Huei-Tse Hou,et al.  Exploring students’ behavioural patterns during online peer assessment from the affective, cognitive, and metacognitive perspectives: a progressive sequential analysis , 2015 .

[7]  Keith Punch,et al.  Introduction to Social Research: Quantitative and Qualitative Approaches , 1998 .

[8]  F. Dochy,et al.  Peer assessment in problem based learning , 2001 .

[9]  Jorge Dı́ez.  Peer Assessment in MOOCs Using Preference Learning via Matrix Factorization , 2013 .

[10]  Bram de Wever,et al.  Structuring the peer assessment process: a multilevel approach for the impact on product improvement and peer feedback quality , 2015, J. Comput. Assist. Learn..

[11]  Erica Sainsbury,et al.  Assessment as a vehicle for learning: extending collaboration into testing , 2008 .

[12]  Randall S. Davies,et al.  Using scaffolded rubrics to improve peer assessment in a MOOC writing course , 2015 .

[13]  N. Falchikov,et al.  Student Peer Assessment in Higher Education: A Meta-Analysis Comparing Peer and Teacher Marks , 2000 .

[14]  Jon M. Pearce,et al.  Enhanced student learning in accounting utilising web-based technology, peer-review feedback and reflective practices: a learning community approach to assessment , 2015 .

[15]  Michael S. Bernstein,et al.  Scaling short-answer grading by combining peer assessment with algorithmic scoring , 2014, L@S.

[16]  Patrick Onghena,et al.  Improving the effectiveness of peer feedback for learning , 2009 .

[17]  Andrew D. Ho,et al.  Changing “Course” , 2014 .

[18]  Carlos Delgado Kloos,et al.  An Algorithm for Peer Review Matching in Massive Courses for Minimising Students' Frustration , 2013, J. Univers. Comput. Sci..

[19]  K. Topping.  Peer Assessment Between Students in Colleges and Universities , 1998 .

[20]  S. Hanrahan,et al.  Assessing Self- and Peer-assessment: The students' views , 2001 .

[21]  Amparo Alonso-Betanzos,et al.  A factorization approach to evaluate open-response assignments in MOOCs using preference learning on peer assessments , 2015, Knowl. Based Syst..

[22]  Stephen P. Balfour,et al.  Assessing Writing in MOOCs: Automated Essay Scoring and Calibrated Peer Review™. , 2013 .

[24]  Young Hoan Cho,et al.  Peer reviewers learn from giving comments , 2011 .

[25]  Allison Littlejohn,et al.  Merlot Journal of Online Learning and Teaching Patterns of Engagement in Connectivist Moocs , 2022 .

[26]  Lino Montoro Moreno,et al.  Student perceptions of peer assessment: an interdisciplinary study , 2014 .

[27]  Nancy Falchikov,et al.  PRODUCT COMPARISONS AND PROCESS BENEFITS OF COLLABORATIVE PEER GROUP AND SELF ASSESSMENTS , 1986 .

[28]  Keith J. Topping,et al.  Methodological quandaries in studying process and outcomes in peer assessment , 2010 .

[29]  Kannan Ramchandran,et al.  A Case for Ordinal Peer-evaluation in MOOCs , 2013 .

[30]  Wilfried Admiraal,et al.  Self- and Peer Assessment in Massive Open Online Courses , 2014 .

[31]  Hoi K. Suen,et al.  Peer assessment for massive open online courses (MOOCs) , 2014 .

[32]  Piotr Mitros,et al.  An Integrated Framework for the Grading of Freeform Responses , 2013 .

[33]  Philip Vickerman,et al.  Student perspectives on formative peer assessment: an attempt to deepen learning? , 2009 .

[34]  C. Reich.  Peer assessment. , 1985, Canadian Medical Association journal.

[35]  Yasemin Demiraslan Çevik,et al.  Assessor or assessee? Investigating the differential effects of online peer assessment roles in the development of students' problem-solving skills , 2015, Comput. Hum. Behav..

[36]  Dorian A. Canelas,et al.  Writing to learn and learning to write across the disciplines: Peer-to-peer writing in introductory-level MOOCs , 2014 .