Reliability and validity of Web-based portfolio peer assessment: A case study for a senior high school's students taking computer course

This study examined the reliability and validity of Web-based portfolio peer assessment. Participants were 72 second-grade students from a senior high school taking a computer course. The results indicated that: 1) there was a lack of consistency across various student raters on a portfolio, or inter-rater reliability; 2) two-thirds of the raters demonstrated inconsistency assessing different portfolios, i.e. inner-rater reliability; 3) peer-assessment scores were not consistent with teacher-assessment scores (criterion-related validity); 4) significant differences were found between peer-assessment scores and end-of-course examination scores, implying that Web-based portfolio peer assessment failed to reflect learning achievements (criterion-related validity). In short, Web-based portfolio peer assessment was not a reliable and valid method.

[1]  Chi-Cheng Chang,et al.  Enhancing self-perceived effects using Web-based portfolio assessment , 2008, Comput. Hum. Behav..

[2]  Chin-Chung Tsai,et al.  Conceptions of and approaches to learning through online peer assessment , 2010 .

[3]  Nicole A. Buzzetto-More,et al.  A Comparative Analysis of Common E-Portfolio Features and Available Platforms , 2007 .

[4]  Chin-Chung Tsai,et al.  University Students’ Perceptions of and Attitudes Toward (Online) Peer Assessment , 2006 .

[5]  Cynthia C Gadbury-Amyot,et al.  Validity and reliability of portfolio assessment of competency in a baccalaureate dental hygiene program. , 2003, Journal of dental education.

[6]  P. Sadler,et al.  The Impact of Self- and Peer-Grading on Student Learning , 2006 .

[7]  P. Miller,et al.  The Effect of Scoring Criteria Specificity on Peer and Self-assessment , 2003 .

[8]  Barry J. Zimmerman,et al.  Developing Self-Regulated Learners: Beyond Achievement to Self-Efficacy , 1996 .

[9]  Aliisa Mylonas,et al.  Developing Procedures for Implementing Peer Assessment in Large Classes Using an Action Research Process , 2002 .

[10]  Dominique Sluijsmans,et al.  Creating a Learning Environment by Using Self-, Peer- and Co-Assessment , 1998 .

[11]  Inci Morgil,et al.  Reliability of Portfolio : A Closer Look at Findings from Recent Publications , 2008 .

[12]  C. Burch,et al.  Inside the Portfolio Experience: The Student's Perspective. , 1999 .

[13]  Sue Bloxham *,et al.  Understanding the rules of the game: marking peer assessment as a medium for developing students' conceptions of assessment , 2004 .

[14]  N. Falchikov,et al.  Student Peer Assessment in Higher Education: A Meta-Analysis Comparing Peer and Teacher Marks , 2000 .

[15]  Chi-Cheng Chang,et al.  Using a Web-based portfolio assessment system to elevate project-based learning performances , 2011, Interact. Learn. Environ..

[16]  Shyan-Ming Yuan,et al.  Developing science activities through a networked peer assessment system , 2002, Comput. Educ..

[17]  Graham Gibbs.  How assessment frames student learning , 2006 .

[18]  Carol Derham,et al.  Digital Professional Portfolios of Preservice Teaching: An Initial Study of Score Reliability and Validity , 2007 .

[19]  Chi-Cheng Chang,et al.  Use and performances of Web-based portfolio assessment , 2009, Br. J. Educ. Technol..

[20]  Christian D. Schunn,et al.  Validity and reliability of scaffolded peer assessment of writing from instructor and student perspectives , 2006 .

[21]  Chi-Cheng Chang.  A study on the evaluation and effectiveness analysis of web-based learning portfolio (WBLP) , 2001, Br. J. Educ. Technol..

[22]  K. Topping.  Peer Assessment Between Students in Colleges and Universities , 1998 .

[23]  Phil Davies,et al.  Computerized Peer Assessment , 2000 .

[24]  Yao-Ting Sung,et al.  Evaluating Proposals for Experiments: An Application of Web-Based Self-Assessment and Peer-Assessment , 2003 .

[25]  James Lenze.  Inter-rater Reliability in the Evaluation of Electronic Portfolios: A Survey of Empirical Research Results , 2004 .

[26]  Chin-Chung Tsai,et al.  On-line peer assessment and the role of the peer feedback: A study of high school computer course , 2007, Comput. Educ..

[27]  F. Prins,et al.  Formative peer assessment in a CSCL environment: a case study , 2005 .

[28]  A. Gélinas,et al.  Issues of reliability and validity in using portfolio assessment to measure foreign language teacher performance , 1998 .

[29]  Michael F. Young,et al.  Reliability and Validity of an Ecologically-Grounded Student Teacher Electronic Portfolio Rubric , 2008 .

[30]  Eric Zhi-Feng Liu,et al.  Web based peer assessment: attitude and achievement , 2001, IEEE Trans. Educ..

[31]  Yao-Ting Sung,et al.  The design and application of a web-based self- and peer-assessment system , 2005, Comput. Educ..

[32]  Alain Jaillet,et al.  Can Online Peer Assessment be Trusted? , 2009, J. Educ. Technol. Soc..

[33]  Robert L. Johnson,et al.  Using Portfolios in Program Evaluation: An Investigation of Interrater Reliability , 2000 .

[34]  Eric Zhi-Feng Liu,et al.  Web-based peer review: the learner as both adapter and reviewer , 2001, IEEE Trans. Educ..

[35]  K. Topping,et al.  Formative Peer Assessment of Academic Writing Between Postgraduate Students , 2000 .

[36]  Chao-hsiu Chen,et al.  The implementation and evaluation of a mobile self- and peer-assessment system , 2010, Comput. Educ..

[37]  Jyh-Chong Liang,et al.  The development of science activities via on-line peer assessment: the role of scientific epistemological views , 2009 .

[38]  M. Jenkins,et al.  Unfulfilled Promise: formative assessment using computer-aidedassessment , 2005 .

[39]  Eric Zhi-Feng Liu,et al.  Web-based peer assessment: feedback for students with various thinking-styles , 2001, J. Comput. Assist. Learn..