Fostering and supporting empirical research on evaluative judgement via a crowdsourced adaptive learning system

The value of students developing the capacity to make accurate judgements about the quality of their work and that of others has been widely recognised in higher education literature. However, despite this recognition, little attention has been paid to the development of tools and strategies with the potential both to foster evaluative judgement and to support empirical research into its growth. This paper provides a demonstration of how educational technologies may be used to fill this gap. In particular, we introduce the adaptive learning system RiPPLE and describe how it aims to (1) develop evaluative judgement in large-class settings through suggested strategies from the literature such as the use of rubrics, exemplars and peer review and (2) enable large empirical studies at low cost to determine the effect-size of such strategies. A case study demonstrating how RiPPLE has been used to achieve these goals in a specific context is presented.

[1]  D. Sadler.  Beyond feedback: developing student capability in complex appraisal , 2010 .

[2]  D. Sadler.  Formative assessment and the design of instructional systems , 1989 .

[3]  Andrew Luxton-Reilly,et al.  A systematic review of tools that support peer assessment , 2009, Comput. Sci. Educ..

[4]  Anastasiya A. Lipnevich,et al.  Mind the gap! Students’ use of exemplars and detailed rubrics as formative assessment , 2014 .

[5]  Dragan Gasevic,et al.  Complementing educational recommender systems with open learner models , 2020, LAK.

[6]  D. Boud,et al.  Using formative assessment to influence self- and co-regulated learning: the role of evaluative judgement , 2018, European Journal of Psychology of Education.

[7]  John Hamer,et al.  PeerWise: students sharing their multiple choice questions , 2008, ICER '08.

[8]  Rebecca Ferguson,et al.  Ethical and privacy issues in the application of learning analytics , 2015, LAK.

[9]  Learning by doing: Do economics students self-evaluation skills improve? , 2017 .

[10]  John Cowan,et al.  Developing the ability for making evaluative judgements , 2010 .

[11]  Simon Bates,et al.  Assessing the quality of a student-generated question repository , 2013, 1308.2202.

[12]  Harald Søndergaard,et al.  Learning from and with peers: the different roles of student peer reviewing , 2009, ITiCSE.

[13]  Christian D. Schunn,et al.  Scaffolded writing and rewriting in the discipline: A web-based reciprocal peer review system , 2007, Comput. Educ..

[14]  Ian Jones,et al.  Peer assessment without assessment criteria , 2014 .

[15]  David Boud,et al.  Does student engagement in self-assessment calibrate their judgement over time? , 2013 .

[16]  H. Andrade,et al.  A review of rubric use in higher education , 2010 .

[17]  Abhir Bhalerao,et al.  Towards electronically assisted peer assessment: a case study , 2001 .

[18]  David Boud,et al.  Reframing assessment as if learning were important , 2007 .

[19]  David Boud,et al.  Developing evaluative judgement: enabling students to make decisions about the quality of work , 2018 .

[20]  Beth Simon,et al.  Quality of student contributed questions using PeerWise , 2009, ACE '09.

[21]  Susan Matlock-Hetzel.  Basic Concepts in Item and Test Analysis. , 1997 .

[22]  D. Boud,et al.  Sustainable assessment revisited , 2016 .

[23]  Ok-Choon Park,et al.  Adaptive Instructional Systems , 2007 .

[24]  Yao-Ting Sung,et al.  The design and application of a web-based self- and peer-assessment system , 2005, Comput. Educ..

[25]  Avril Thomson,et al.  Rethinking feedback practices in higher education: a peer review perspective , 2014 .

[26]  Mark Toleman,et al.  Electronic peer review: a large cohort teaching themselves? , 2005 .

[27]  Dragan Gasevic,et al.  A Multivariate ELO-based Learner Model for Adaptive Educational Systems , 2019, EDM.

[28]  Rebecca Ferguson,et al.  Setting learning analytics in context: overcoming the barriers to large-scale adoption , 2014, LAK '14.

[29]  D. Carless,et al.  Developing students’ capacities for evaluative judgement through analysing exemplars , 2018 .

[30]  George Siemens,et al.  Ethical and privacy principles for learning analytics , 2014, Br. J. Educ. Technol..

[31]  Gail M Sullivan,et al.  Getting off the "gold standard": randomized controlled trials and education research. , 2011, Journal of graduate medical education.

[32]  Mohammed K Khalil,et al.  Applying learning theories and instructional design models for effective instruction. , 2016, Advances in physiology education.

[33]  David Nicol,et al.  Guiding Principles for Peer Review:Unlocking Learners’ Evaluative Skills , 2014 .

[34]  Jacob Whitehill,et al.  Do Learners Know What's Good for Them? Crowdsourcing Subjective Ratings of OERs to Predict Learning Gains , 2019, EDM.

[35]  Linda Corrin,et al.  The Ethics of Learning Analytics in Australian Higher Education: Discussion Paper , 2019 .

[36]  Steve Joordens,et al.  Peering into large lectures: examining peer and expert mark agreement using peerScholar, an online peer assessment tool , 2008, J. Comput. Assist. Learn..

[37]  Andrew Luxton-Reilly,et al.  Rubrics used in peer assessment , 2016, ACSW.

[38]  D. Royce Sadler,et al.  Three in-course assessment reforms to improve higher education learning outcomes , 2016 .

[39]  D. Rubin,et al.  The central role of the propensity score in observational studies for causal effects , 1983 .

[40]  Fiona Spence,et al.  Peer assessment using aropä , 2007 .

[41]  Beryl Plimmer,et al.  StudySieve: a tool that supports constructive evaluation for free-response questions , 2010, CHINZ '10.

[42]  Dragan Gasevic,et al.  Development and Adoption of an Adaptive Learning System: Reflections and Lessons Learned , 2020, SIGCSE.

[43]  Neil T. Heffernan,et al.  The Future of Adaptive Learning: Does the Crowd Hold the Key? , 2016, International Journal of Artificial Intelligence in Education.

[44]  Abhir Bhalerao,et al.  Towards electronically assisted a case study , 2011 .

[45]  Carole Torgerson,et al.  Randomised controlled trials (RCTs) in education research –methodological debates, questions, challenges , 2018, Educational Research.

[46]  Nea Pirttinen,et al.  Analysis of Students' Peer Reviews to Crowdsourced Programming Assignments , 2018, Koli Calling.

[47]  Joseph Jay Williams,et al.  RiPPLE: A Crowdsourced Adaptive Platform for Recommendation of Learning Activities , 2019, J. Learn. Anal..