Theory-driven Design of an IT-based Peer Assessment to Assess High Cognitive Levels of Educational Objectives in Large-scale Learning Services

Due to increasing numbers of students at German universities large-scale learning services are still a common default. These learning services lack interaction as well as feedback to assess learners’ knowledge on the cognitive levels of educational objectives. This situation is alarming, since interaction and feedback in order to assess the own learning progress are important factors for individual learning success and satisfaction. The use of an IT-based peer assessment as a learning instrument can help overcome these challenges by increasing interaction and feedback without massively increasing the workload of lecturers. In this research-in-progress paper we present a theory-driven design of an IT-based peer assessment aiming to increase interaction and feedback as well as assess learners’ knowledge on high cognitive levels of educational objectives in large-scale learning services. We fol- low a design science research approach and rely on insights from theory of interaction and feedback in order to gather requirements as well as derive design elements to create the IT-based peer assessment. As a next step, we will use the instrument in our large-scale learning service aiming to evaluate whether the IT-based peer assessment is useful to assess high cognitive levels of educational objectives, hence supporting learners during their learning process.

[1]  D. Sluijsmans,et al.  Effective peer assessment processes: Research findings and future directions , 2010 .

[2]  N. Hara.  STUDENT DISTRESS IN A WEB-BASED DISTANCE EDUCATION COURSE , 2000 .

[3]  W. Cheng,et al.  Having second thoughts: Student perceptions before and after a peer assessment exercise , 1997 .

[4]  Horst Otto Mayer,et al.  Lernzielüberprüfung im eLearning , 2009 .

[5]  Peggy A. Ertmer,et al.  The expert learner: Strategic, self-regulated, and reflective , 1996 .

[6]  Doiron.  The Value of Online Student Peer Review, Evaluation and Feedback in Higher Education , 2003 .

[7]  Matthias Söllner,et al.  Theory-Driven Design of a Mobile-Learning Application to Support Different Interaction Types in Large-Scale Lectures , 2014, ECIS.

[8]  Jan Marco.  THEORY-DRIVEN DESIGN OF AN IT-BASED PEER ASSESSMENT TO ASSESS HIGH COGNITIVE LEVELS OF EDUCATIONAL OBJECTIVES IN LARGE-SCALE LEARNING SERVICES , 2015 .

[9]  Rod Sims,et al.  Promises of Interactivity: Aligning Learner Perceptions and Expectations With Strategies for Flexible and Online Learning , 2003 .

[10]  Andreas Janson,et al.  Flipping the IS Classroom - Theory-Driven Design for Large-Scale Lectures , 2014, ICIS.

[11]  Robert O. Briggs,et al.  On theory-driven design and deployment of collaboration systems , 2006, Int. J. Hum. Comput. Stud..

[12]  Irda Haryani Tahir.  A Study on Peer Evaluation and Its Influence on College ESL Students , 2012 .

[13]  L. Darling-Hammond,et al.  Authentic assessment in action : studies of schools and students at work , 1995 .

[14]  Irfan Naufal Umar,et al.  Learning Style, Metaphor and Pair Programming: Do they Influence Performance? , 2012 .

[15]  Gerald Eisenkopf,et al.  Peer effects, motivation, and learning , 2010 .

[16]  F. Paas,et al.  Effects of elicited reflections combined with tutor or peer feedback on self-regulated learning and learning outcomes , 2007 .

[17]  W S Maki,et al.  Evaluation of a Web-based introductory psychology course: I. Learning and satisfaction in on-line versus lecture courses , 2000, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[18]  Yao-Ting Sung,et al.  The design and application of a web-based self- and peer-assessment system , 2005, Comput. Educ..

[19]  Christopher Rushton,et al.  Peer Assessment in a Collaborative Hypermedia Environment: A Case Study. , 1993 .

[20]  M. Moore.  Editorial: Three types of interaction , 1989 .

[21]  Randall W. Sadler,et al.  The effect and affect of peer review in electronic versus traditional modes on L2 writing , 2003 .

[22]  Birgit Leidenfrost,et al.  Verbesserung der Studiensituation für StudienanfängerInnen durch Cascaded Blended Mentoring , 2009 .

[23]  Jerry Andriessen,et al.  Learning through synchronous electronic discussion , 2000, Comput. Educ..

[24]  Fernando Alonso,et al.  A moderate constructivist e-learning instructional model evaluated on computer specialists , 2009, Comput. Educ..

[25]  David Boud,et al.  Self and peer assessment in professional education: a preliminary study , 1979 .

[26]  Frederick K. S. Leung,et al.  A study of the alignment of learning targets and assessment to generic skills in the new senior secondary mathematics curriculum in Hong Kong , 2014 .

[27]  D. Sluijsmans,et al.  Fostering Interactivity through Formative Peer Assessment in (Web-Based) Collaborative Learning Envi , 2009 .

[28]  Frank Tuzi,et al.  The impact of e-feedback on the revisions of L2 writers in an academic writing course , 2004 .

[29]  D. Boud,et al.  Rethinking assessment in higher education : learning for the longer term , 2007 .

[30]  P. Sadler,et al.  The Impact of Self- and Peer-Grading on Student Learning , 2006 .

[31]  Justin Cheng,et al.  Peer and self assessment in massive online classes , 2013, ACM Trans. Comput. Hum. Interact..

[32]  K. Topping.  Trends in Peer Learning , 2005 .

[33]  Richard E. Mayer,et al.  Learning by viewing versus learning by doing: Evidence-based guidelines for principled learning environments , 2008 .

[34]  Chin-Chung Tsai,et al.  On-line peer assessment and the role of the peer feedback: A study of high school computer course , 2007, Comput. Educ..

[35]  Zane L. Berge,et al.  Creating Student Interaction within the Educational Experience: A Challenge for Online Teachers , 2009 .

[36]  Alain Jaillet,et al.  Can Online Peer Assessment be Trusted? , 2009, J. Educ. Technol. Soc..

[37]  Deborah L. Butler,et al.  Structuring Instruction to Promote Self-Regulated Learning by Adolescents and Adults With Learning Disabilities , 2003 .

[38]  Martin Bichler,et al.  Design science in information systems research , 2006, Wirtschaftsinf..

[39]  Benjamin S. Bloom,et al.  A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives , 2000 .

[40]  Nancy Sullivan,et al.  A comparative study of two ESL writing environments: A computer-assisted classroom and a traditional oral classroom , 1996 .

[41]  Jan Marco Leimeister,et al.  Der Wert von IT-gestütztem Peer Assessment zur Unterstützung des Lernens in einer Universitären Massenlehrveranstaltung , 2015, Wirtschaftsinformatik.

[42]  Gordon B. Davis,et al.  User Acceptance of Information Technology: Toward a Unified View , 2003, MIS Q..

[43]  Urban Nuldén,et al.  ‘Copernicus’: Experiencing a Failing Project for Reflection and Learning , 2005 .

[44]  Diane M. Christophel,et al.  College teacher immediacy and student ratings of instruction , 1996 .

[45]  D. Sluijsmans,et al.  The use of self-, peer and co-assessment in higher education: A review , 1999 .

[46]  Tzu-Chien Liu,et al.  Embedding EduClick in Classroom to Enhance Interaction , 2004 .

[47]  Zhenghao Chen,et al.  Tuned Models of Peer Assessment in MOOCs , 2013, EDM.

[48]  Ian Mugridge.  Growth and Structure of Distance Education , 1987 .

[49]  Yvonne Steinert,et al.  Interactive lecturing: strategies for increasing participation in large group presentations , 1999 .

[50]  Andrä Wolter,et al.  Studiennachfrage, Absolventenverbleib und Fachkräftediskurs – Wohin steuert die Hochschulentwicklung in Deutschland? , 2014 .

[51]  Sean B. Eom,et al.  The Determinants of Students' Perceived Learning Outcomes and Satisfaction in University Online Education: An Empirical Investigation* , 2006 .

[52]  Mike Moore,et al.  Distance Education: A Systems View , 1995 .

[53]  Andreas Janson,et al.  Flipping the Classroom – IT-unterstützte Lerneraktivierung zur Verbesserung des Lernerfolges einer universitären Massenlehrveranstaltung , 2015, HMD Praxis der Wirtschaftsinformatik.

[54]  Laurel Evelyn Dyson,et al.  Interactive classroom mLearning and the experiential transactions between students and lecturer , 2009 .

[55]  Erhan Delen,et al.  Effects of interactivity and instructional scaffolding on learning: Self-regulation in online video-based environments , 2014, Comput. Educ..

[56]  H. Walberg,et al.  What Influences Learning? A Content Analysis of Review Literature , 1990 .

[57]  E. Gagné.  The cognitive psychology of school learning , 1985 .

[58]  N. Falchikov,et al.  Student Peer Assessment in Higher Education: A Meta-Analysis Comparing Peer and Teacher Marks , 2000 .

[59]  Robert P. Bostrom,et al.  Technology-Mediated Learning: A Comprehensive Theoretical Model , 2009, J. Assoc. Inf. Syst..

[60]  Youngjin Yoo,et al.  A Comparative Study of Distributed Learning Environments on Learning Outcomes , 2002, Inf. Syst. Res..