A Novel Technique for Comparing the Reliability of Multiple Peer Assessments with that of Single Teacher Assessments of Group Process Work

There have been no studies which compare the reliabilities of peer and teacher ratings of group process work. This is mainly because there is usually just one teacher who is in the position to rate the work of a group over time. Hence it is not possible in these circumstances to determine the reliability of teacher ratings. The reliability of multiple peer ratings can be estimated, however, through traditional analysis of variance (ANOVA) methods. This paper employs a novel application of ANOVA techniques to compare the reliability of multiple peer ratings with single tutor ratings. The method is outlined and demonstrated by application to rating data from two different courses. Both involved sets of multiple peer and individual tutor ratings from parallel classes that were used to assess student contributions to group process work. Implications for practitioners and researchers are discussed in terms both of the application of the technique and the findings from the two case studies.

[1]  Walter M. Houston,et al.  Adjustments for Rater Effects in Performance Assessment , 1991 .

[2]  B. J. Winer.  Statistical Analysis in Psychology and Education. 3rd ed. , 1972 .

[3]  G. Rowley.  Notes and Comments: The Reliability of Observational Measures , 1976 .

[4]  I. E. Hughes,et al.  Staff and peer-group assessment of oral communication skills , 1993 .

[5]  G. Norman,et al.  Pitfalls in the pursuit of objectivity: issues of reliability , 1991, Medical education.

[6]  B. Montgomery,et al.  An Interactionist Analysis of Small Group Peer Assessment , 1986 .

[7]  R. Rosati,et al.  Reliability and Accuracy of Resident Evaluations of Surgical Faculty , 1992, Evaluation & the health professions.

[8]  Mark Freeman,et al.  Peer Assessment by Groups of Group Work , 1995 .

[9]  Quinn McNemar,et al.  Statistical Analysis in Psychology and Education. , 1967 .

[10]  T. Marteau,et al.  Factors influencing the communication skills of first‐year clinical medical students , 1991, Medical education.

[11]  Graham Mowl,et al.  Using Self and Peer Assessment to Improve Students’ Essay Writing: a Case Study from Geography , 1995 .

[12]  Nancy Falchikov,et al.  Self and Peer Assessment of a Group Project Designed to Promote the Skills of Capability. , 1988 .

[13]  Glenn L. Rowley,et al.  The Reliability of Observational Measures. , 1976 .

[14]  N. Falchikov.  Peer Feedback Marking: Developing Peer Assessment , 1995 .

[15]  D. Boud,et al.  Quantitative studies of student self-assessment in higher education: a critical analysis of findings , 1989 .

[16]  B. Linn,et al.  Performance rating scale for peer and self assessment 1 , 1975, British journal of medical education.

[17]  R. L. Ebel,et al.  Estimation of the reliability of ratings , 1951 .

[18]  K. Topping.  Peer Assessment Between Students in Colleges and Universities , 1998 .

[19]  Francis Lopez-Real,et al.  Peer Assessment of a Group Project in a Primary Mathematics Education Course , 1999 .

[20]  L. Stefani.  Peer, self and tutor assessment: Relative reliabilities , 1994 .