The Comprehensive Assessment of Team Member Development of a Behaviorally Anchored Rating Scale for Self- and Peer Evaluation

Instructors often incorporate selfand peer evaluations when they use teamwork in their classes, which is common in management education. However, the process is often time consuming and frequently does not match well with guidance provided by the literature. We describe the development of a web-based instrument that efficiently collects and analyzes selfand peer-evaluation data. The instrument uses a behaviorally anchored rating scale to measure team-member contributions in five areas based on the team effectiveness literature. Three studies provide evidence for the validity of the new instrument. Implications for management education and areas for future research are discussed.

[1]  Stéphane Brutus,et al.  Can Business Students Learn to Evaluate Better? Evidence From Repeated Exposure to a Peer-Evaluation System , 2013 .

[2]  Subhash Sharma,et al.  Assessing generalizability of scales used in cross-national research , 2003 .

[3]  V. Druskat,et al.  Effects and timing of developmental peer appraisals in self-managing work groups. , 1999, The Journal of applied psychology.

[4]  George C. Thornton,et al.  Construct Validity of Self- and Peer Evaluations of Performance Dimensions in an Assessment Center , 1992 .

[5]  Matthew W. Ohland,et al.  Design and Validation of a Web-Based System for Assigning Members to Teams Using Instructor-Specified Criteria , 2010 .

[6]  Lorne M. Sulsky,et al.  Rating Formats and Rater Training Redux: A Context-Specific Approach for Enhancing the Effectiveness of Performance Management , 2009 .

[7]  Steven K. Jones,et al.  Developing and assessing college student teamwork skills , 2011 .

[8]  Barbara J. Millis,et al.  Cooperative Learning for Higher Education Faculty , 1997 .

[9]  Gary L. May,et al.  Collaborative Peer Evaluation: Best Practices for Group Member Assessments , 2002 .

[10]  Janet L. Barnes-Farrell,et al.  Components of Teamwork: Impact on Evaluations of Contributions to Work Team Effectiveness1 , 1997 .

[11]  R. Brennan.  Performance Assessments from the Perspective of Generalizability Theory , 2000 .

[12]  G. Neuman,et al.  Team effectiveness: beyond skills and cognitive ability. , 1999, The Journal of applied psychology.

[13]  Jack McGourty.  The Team Developer: An Assessment and Skill Building Program , 2000 .

[14]  Donald B. Fedor,et al.  Peer and Upward Appraisals , 1997 .

[15]  R. Saavedra,et al.  Peer evaluation in self-managing work groups. , 1993 .

[16]  S. Taggar,et al.  Interpersonal Affect and Peer Rating Bias in Teams , 2006 .

[17]  Robert C. Harris,et al.  Undergraduate Marketing Students, Group Projects, and Teamwork: The Good, the Bad, and the Ugly? , 1999 .

[18]  Murray R. Barrick,et al.  Relating member ability and personality to work-team processes and team effectiveness. , 1998 .

[19]  L. Willcoxson.  “It’s not Fair!”: Assessing the Dynamics and Resourcing of Teamwork , 2006 .

[20]  Chockalingam Viswesvaran,et al.  Comparative analysis of the reliability of job performance ratings , 1996 .

[21]  Donald B. Rubin,et al.  The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles. , 1974 .

[22]  R. E. Allen,et al.  Managerial Discretion in the Use Of Self-Ratings in an Appraisal System: The Antecedents And Consequences , 2004 .

[23]  James W. Smither,et al.  Accountability: The Achilles' Heel of Multisource Feedback , 1997 .

[24]  Richard J. Klimoski,et al.  Training Undergraduates to Work in Organizational Teams , 2004 .

[25]  Bo Zhang,et al.  How to Assign Individualized Scores on a Group Project: An Empirical Evaluation , 2009 .

[26]  Nigel K. L. Pope,et al.  The impact of stress in self‐ and peer assessment , 2005 .

[27]  Jane Burdett,et al.  Predicting Satisfaction with Group Work Assignments , 2009, Journal of University Teaching and Learning Practice.

[28]  Stéphane Brutus,et al.  Improving the Effectiveness of Students in Groups With a Centralized Peer Evaluation System , 2010 .

[29]  Dana E. Sims,et al.  Is there a “Big Five” in Teamwork? , 2005 .

[30]  David J. Woehr,et al.  Rater training for performance appraisal: A quantitative review , 1994 .

[31]  Charles M. Brooks,et al.  Free Riding in Group Projects and the Effects of Timing, Frequency, and Specificity of Criteria in Peer Assessments , 2003 .

[32]  Patricia Ann Mabrouk,et al.  Active learning : models from the analytical sciences , 2007 .

[33]  Kenneth J. Chapman,et al.  Creating Positive Group Project Experiences: An Examination of the Role of the Instructor on Students’ Perceptions of Group Projects , 2001 .

[34]  Chockalingam Viswesvaran,et al.  Is there a general factor in ratings of job performance? A meta-analytic framework for disentangling substantive and error influences. , 2005, The Journal of applied psychology.

[35]  Jay McCormack,et al.  Assessing team member citizenship in capstone engineering design courses , 2010 .

[36]  Jane Burdett,et al.  Making Groups Work: University Students' Perceptions , 2003 .

[37]  Martin R. Fellenz.  Toward Fairness in Assessing Student Groupwork: A Protocol for Peer Evaluation of Individual Contributions , 2006 .

[38]  James R. Bailey,et al.  The Relative Contribution of Emotional Competence and Cognitive Ability to Individual and Team Performance , 2004, Emotion and Performance.

[39]  Matthew W. Ohland,et al.  Effects of Behavioral Anchors on Peer Evaluation Reliability , 2005 .

[40]  Randall S. Hansen,et al.  Benefits and Problems With Student Teams: Suggestions for Improving Team Projects , 2006 .

[41]  J. Kruger,et al.  Unskilled and unaware of it: how difficulties in recognizing one's own incompetence lead to inflated self-assessments. , 1999, Journal of personality and social psychology.

[42]  Ann Walker,et al.  British Psychology Students' Perceptions of Group-Work and Peer Assessment , 2001 .

[43]  E. Mannix,et al.  The Dynamic Nature of Conflict: A Longitudinal Study of Intragroup Conflict and Group Performance. , 2001 .

[44]  R. Rosenstein.  The Teamwork Components Model: An Analysis Using Structural Equation Modeling , 1994 .

[45]  Flora P. McMartin,et al.  Methods to improve the validity and sensitivity of a self/peer assessment instrument , 2000, IEEE Trans. Educ..

[46]  D. Sluijsmans,et al.  The use of self-, peer and co-assessment in higher education: A review , 1999 .

[47]  Richard M. Felder,et al.  Accounting for Individual Effort in Cooperative Learning Teams , 2000 .

[48]  D. Loyd,et al.  Classroom research: Bridging the ivory divide , 2005 .

[49]  Patricia Huddleston,et al.  Does It Matter if I Hate Teamwork? What Impacts Student Attitudes toward Teamwork , 2003 .

[50]  J. Raelin,et al.  Does Action Learning Promote Collaborative Leadership? , 2006 .

[51]  David J. Woehr,et al.  Understanding frame-of-reference training: the impact of training on the recall of performance information , 1994 .

[52]  J. Hackman,et al.  The design of work teams , 1987 .

[53]  M. Brucker,et al.  Playing well with others. , 2014, Nursing for women's health.

[54]  Avan R. Jassawalla,et al.  Students’ Perceptions of Social Loafing: Its Antecedents and Consequences in Undergraduate Business Classroom Teams , 2009 .

[55]  Qiang Lu,et al.  Measuring member performance in multi-functional R&D teams: an empirical study with GAHP analysis , 2010, 2010 IEEE International Conference on Industrial Engineering and Engineering Management.

[56]  Imad H. Elhajj,et al.  Turning Student Groups into Effective Teams , 2004 .

[57]  John Schaubroeck,et al.  A meta-analysis of self-supervisor, self-peer, and peer-supervisor ratings. , 1988 .

[58]  Alain Fayolle,et al.  Tangling With Spaghetti: Pedagogical Lessons From Games , 2009 .

[59]  Yining Chen,et al.  Students' Perceptions of Peer Evaluation: An Expectancy Perspective , 2004 .

[60]  P. Aggarwal,et al.  Social Loafing on Group Projects , 2008 .

[61]  Peter A. Bamberger,et al.  Peer Assessment, Individual Performance, and Contribution to Group Processes , 2005 .

[62]  Matthew W. Ohland,et al.  Development of a Theory-Based Assessment of Team Member Effectiveness , 2007 .

[63]  D. Baker.  Peer Assessment in Small Groups: A Comparison of Methods , 2008 .

[64]  Ou Lydia Liu,et al.  Assessing Teamwork and Collaboration in High School Students , 2009 .

[65]  Glyn Thomas,et al.  Using self- and peer-assessment to enhance students’ future-learning in higher education. , 2011, Journal of University Teaching and Learning Practice.

[66]  Kim A. Stewart,et al.  Lessons from the Best and Worst Student Team Experiences: How a Teacher can make the Difference , 1999 .

[67]  Michael P. Kirsch,et al.  METAANALYSES OF VALIDITY STUDIES PUBLISHED BETWEEN 1964 AND 1982 AND THE INVESTIGATION OF STUDY CHARACTERISTICS , 1984 .

[68]  Carol Bormann Young,et al.  Viewpoint: A Conceptual Framework for Designing Group Projects , 2000 .

[69]  A. Paswan,et al.  Dimensions of Peer Evaluation, Overall Satisfaction, and Overall Evaluation: An Investigation in a Group Task Environment , 2004 .

[70]  D. Kolb,et al.  Learning Styles and Learning Spaces: Enhancing Experiential Learning in Higher Education , 2005 .

[71]  S. Sonnentag,et al.  Individual-Level Predictors of Task-Related Teamwork Processes , 2009 .

[72]  Allen I. Huffcutt,et al.  Psychometric properties of multisource performance ratings: A meta-analysis of subordinate, supervisor, peer, and self-ratings. , 1997 .

[73]  Travor C. Brown,et al.  Problem-Solving Team Behaviors , 2001 .

[74]  Laurie R. Weingart,et al.  Innovation in an Academic Setting: Designing and Leading a Business Through Market Focused, Interdisciplinary Teams ARTHUR A. BONI , 2009 .

[75]  M. A. Campion,et al.  The Knowledge, Skill, and Ability Requirements for Teamwork: Implications for Human Resource Management , 1994 .

[76]  Michael J. Stevens,et al.  Staffing Work Teams: Development and Validation of a Selection Test for Teamwork Settings , 1999 .

[77]  Eva Nick,et al.  The dependability of behavioral measurements: theory of generalizability for scores and profiles , 1973 .

[78]  R. Hooijberg,et al.  Using Multisource Feedback Coaching Effectively in Executive Education , 2009 .

[79]  Joan Ruelle,et al.  Playing Well with Others , 2001 .

[80]  P. C. Smith,et al.  Retranslation of expectations: An approach to the construction of unambiguous anchors for rating scales. , 1963 .

[81]  T. Gatfield.  Examining Student Satisfaction with Group Projects and Peer Assessment , 1999 .

[82]  Bianey Cristina Ruiz Ulloa,et al.  Attitude toward teamwork and effective teaming , 2004 .

[83]  A. Erez,et al.  EFFECTS OF ROTATED LEADERSHIP AND PEER EVALUATION ON THE FUNCTIONING AND EFFECTIVENESS OF SELF‐MANAGED TEAMS: A QUASI‐EXPERIMENT , 2002 .

[84]  M. Bell.  From the Editors What if … ?: Diversity, Scholarship, and Impact , 2010 .

[85]  Rebecca M. Guidice,et al.  Accountability in a Performance Appraisal Context: The Effect of Audience and Form of Accounting on Rater Response and Behavior , 2007 .

[86]  Marvin D. Dunnette,et al.  The development and evaluation of behaviorally based rating scales , 1973 .

[87]  P. Dominick,et al.  The Effects of Peer Feedback on Team Member Behavior , 1997 .

[88]  L. Miles,et al.  Assessing contributions to group assignments , 2004 .

[89]  David C. Sharp,et al.  More Than Fun and Games: Reconsidering the Virtues of Strategic Management Simulations , 2005 .

[90]  R. Brennan.  Variance Components in Generalizability Theory , 1994 .

[91]  P. B. Hess.  Enhancing Leadership Skill Development By Creating Practice/Feedback Opportunities in the Classroom , 2007 .