Developing a Peer Evaluation Instrument that is Simple, Reliable, and Valid

A multi-university research team is working to design a peer evaluation instrument for cooperative learning teams that is simple, reliable, and valid. In this work, an overview of the process of developing behaviorally anchored rating scales (BARS) will be presented, including the establishment of a theoretical basis for the instrument and a description of the extensive classroom testing of the draft instrument conducted during fall 2004. Introducing the draft instrument to the engineering education community through exposure in the NSF grantees' poster session is expected both to improve the validity of the scale itself through the feedback we receive and to accelerate the dissemination of the instrument.

[1]  Richard M. Felder,et al.  Peer Ratings in Cooperative Learning Teams , 1999 .

[2]  D. Grasso,et al.  Socially‐Relevant Design: The TOYtech Project at Smith College , 2002 .

[3]  Barbara J. Millis,et al.  Cooperative Learning for Higher Education Faculty , 1997 .

[4]  J. McGrath.  Social Psychology: A Brief Introduction , 1964 .

[5]  A. Howard.  The changing nature of work , 1995 .

[6]  A. Erez,et al.  The nature and dimensionality of organizational citizenship behavior: a critical review and meta-analysis. , 2002, The Journal of applied psychology.

[7]  Flora P. McMartin,et al.  Methods to improve the validity and sensitivity of a self/peer assessment instrument , 2000, IEEE Trans. Educ..

[8]  M. Ohland,et al.  Peer Evaluation In A Mandatory Cooperative Education Environment , 2001 .

[9]  Richard M. Felder,et al.  Accounting for Individual Effort in Cooperative Learning Teams , 2000 .

[10]  E. Seat,et al.  Administering, scoring and debriefing team developer , 2001, 31st Annual Frontiers in Education Conference. Impact on Engineering and Science Education. Conference Proceedings (Cat. No.01CH37193).

[11]  Jack McGourty.  The Team Developer: An Assessment and Skill Building Program , 2000 .

[12]  L. Crocker,et al.  Introduction to Classical and Modern Test Theory , 1986 .

[13]  J. Hackman.  Group influences on individuals in organizations. , 1992 .

[14]  Michael J. Stevens,et al.  Staffing Work Teams: Development and Validation of a Selection Test for Teamwork Settings , 1999 .

[15]  R. Felder,et al.  Learning and Teaching Styles in Engineering Education. , 1988 .

[16]  Cynthia J. Finelli.  Assessing improvement in students' team skills and using a learning style inventory to increase it , 2001, 31st Annual Frontiers in Education Conference. Impact on Engineering and Science Education. Conference Proceedings (Cat. No.01CH37193).

[17]  J Noak,et al.  Performance rating. , 1999, Nursing standard (Royal College of Nursing (Great Britain) : 1987).

[18]  J. Salgado.  The Five Factor Model of personality and job performance in the European Community. , 1997, The Journal of applied psychology.

[19]  D. Ilgen,et al.  The Changing Nature of Performance: Implications for Staffing, Motivation, and Development. Frontiers of Industrial and Organizational Psychology. , 1999 .

[20]  H. Demsetz,et al.  Production, Information Costs, and Economic Organization , 1975, IEEE Engineering Management Review.

[21]  P. Wankat.  An Analysis of the Articles in the Journal of Engineering Education , 1999 .

[22]  J. McGourty,et al.  Developing a Comprehensive Assessment Program for Engineering Education * , 1998 .

[23]  S. Rothmann,et al.  THE BIG FIVE PERSONALITY DIMENSIONS AND JOB PERFORMANCE , 2003 .

[24]  J. P. Near,et al.  ORGANIZATIONAL CITIZENSHIP BEHAVIOR: ITS NATURE AND ANTECEDENTS , 1983 .

[25]  A. H. Church,et al.  Advancing the State of the Art of 360-Degree Feedback , 1997 .

[26]  Jack McGourty.  Using multisource feedback in the classroom: a computer-based approach , 2000, IEEE Trans. Educ..

[27]  Murray R. Barrick,et al.  THE BIG FIVE PERSONALITY DIMENSIONS AND JOB PERFORMANCE: A META-ANALYSIS , 1991 .

[28]  David W. Johnson,et al.  Active Learning: Cooperation in the College Classroom , 2006 .

[29]  Richard A. Guzzo,et al.  Group performance and intergroup relations in organizations. , 1992 .

[30]  M. D. Dunnette.  Handbook of Industrial and Organizational Psychology , 2005 .

[31]  Eduardo Salas,et al.  Team Effectiveness and Decision Making in Organizations , 1995 .

[32]  Matthew W. Ohland,et al.  Comparing the Reliability of Two Peer Evaluation Instruments , 2000 .

[33]  R. S. Thompson.  Relative validity of peer and self-evaluations in self-directed interdependent work teams , 2001, 31st Annual Frontiers in Education Conference. Impact on Engineering and Science Education. Conference Proceedings (Cat. No.01CH37193).

[34]  W. Borman,et al.  A Theory of Individual Differences in Task and Contextual Performance , 1997 .

[35]  Michael J. Miller,et al.  Distributional Ratings of Performance: More Evidence for a New Rating Format , 1997 .

[36]  Matthew W. Ohland,et al.  Peer Evaluations In Teams Of Predominantly Minority Students , 2000 .

[37]  R. W. Brown.  Autorating: getting individual marks from team marks and enhancing teamwork , 1995, Proceedings Frontiers in Education 1995 25th Annual Conference. Engineering Education for the 21st Century.

[38]  Gary D. Borich,et al.  Educational Testing and Measurement : Classroom Application and Practice , 1984 .

[39]  P. C. Smith,et al.  Retranslation of expectations: An approach to the construction of unambiguous anchors for rating scales. , 1963 .

[40]  R. Robinson.  Calibrated Peer Review™ , 2009 .