Workplace-based assessment: raters’ performance theories and constructs

Weaknesses in the nature of rater judgments are generally considered to compromise the utility of workplace-based assessment (WBA). In order to gain insight into the underpinnings of rater behaviours, we investigated how raters form impressions of and make judgments on trainee performance. Using theoretical frameworks of social cognition and person perception, we explored raters’ implicit performance theories, use of task-specific performance schemas and the formation of person schemas during WBA. We used think-aloud procedures and verbal protocol analysis to investigate schema-based processing by experienced (N = 18) and inexperienced (N = 16) raters (supervisor-raters in general practice residency training). Qualitative data analysis was used to explore schema content and usage. We quantitatively assessed rater idiosyncrasy in the use of performance schemas and we investigated effects of rater expertise on the use of (task-specific) performance schemas. Raters used different schemas in judging trainee performance. We developed a normative performance theory comprising seventeen inter-related performance dimensions. Levels of rater idiosyncrasy were substantial and unrelated to rater expertise. Experienced raters made significantly more use of task-specific performance schemas compared to inexperienced raters, suggesting more differentiated performance schemas in experienced raters. Most raters started to develop person schemas the moment they began to observe trainee performance. The findings further our understanding of processes underpinning judgment and decision making in WBA. Raters make and justify judgments based on personal theories and performance constructs. Raters’ information processing seems to be affected by differences in rater expertise. The results of this study can help to improve rater training, the design of assessment instruments and decision making in WBA.

[1]  Pamela A. Moss,et al.  Can There Be Validity Without Reliability? , 1994 .

[2]  Arianne Heinrichs.  Chromatin: A case of the tail wagging the dog , 2006, Nature Reviews Molecular Cell Biology.

[3]  E. Holmboe,et al.  Perspective: the ACGME toolbox: half empty or half full? , 2010, Academic medicine : journal of the Association of American Medical Colleges.

[4]  Karol G. Ross,et al.  The Cambridge Handbook of Expertise and Expert Performance: Professional Judgments and “Naturalistic Decision Making” , 2006 .

[5]  H. Saedon,et al.  Workplace-based assessment as an educational tool . , 2022 .

[6]  Theresa J. B. Kline,et al.  Understanding Frame-of-Reference Training Success: A Social Learning Theory Perspective , 2007 .

[7]  M. Donnelly,et al.  Factors in faculty evaluation of medical students' performance , 2007, Medical education.

[8]  D. A. Kenny,et al.  The how and why of disagreement among perceivers: An exploration of person models , 2006 .

[9]  Marjan J. B. Govaerts,et al.  Broadening Perspectives on Clinical Performance Assessment: Rethinking the Nature of In-training Assessment , 2007, Advances in health sciences education : theory and practice.

[10]  Kevin R. Murphy,et al.  Understanding Performance Appraisal: Social, Organizational, and Goal-Based Perspectives , 1995 .

[11]  V. Pankratz,et al.  Internal structure of mini-CEX scores for internal medicine residents: factor analysis and generalizability , 2010, Advances in health sciences education : theory and practice.

[12]  D. Ilgen,et al.  Cognitive categories of raters and rating accuracy , 1992 .

[13]  E. Holmboe.  Direct Observation by Faculty , 2007 .

[14]  Eric S. Holmboe,et al.  Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. , 2009, JAMA.

[15]  Paul J. Feltovich,et al.  The Cambridge handbook of expertise and expert performance , 2006 .

[16]  Eliot R. Smith,et al.  Contextualizing person perception: distributed social cognition. , 2009, Psychological review.

[17]  R. Power,et al.  Checklists for improving rigour in qualitative research , 2001, BMJ : British Medical Journal.

[18]  R. Cardy,et al.  The effects of individual performance schemata and dimension familiarization on rating accuracy , 1987 .

[19]  G. Regehr,et al.  Toward Authentic Clinical Evaluation: Pitfalls in the Pursuit of Competency , 2010, Academic medicine : journal of the Association of American Medical Colleges.

[20]  Steven E. Scullen,et al.  Understanding the latent structure of job performance ratings. , 2000, The Journal of applied psychology.

[21]  C. Neil Macrae,et al.  Social cognition: Categorical person perception. , 2001, British journal of psychology.

[22]  Causes and consequences of stereotypes in organizations. , 2013 .

[23]  J Noak,et al.  Performance rating. , 1999, Nursing standard (Royal College of Nursing (Great Britain) : 1987).

[24]  M. Chi.  Quantifying Qualitative Analyses of Verbal Data: A Practical Guide , 1997 .

[25]  C. V. D. van der Vleuten,et al.  Characteristics of communication guidelines that facilitate or impede guideline use: a focus group study , 2007, BMC family practice.

[26]  J. Norcini.  Current perspectives in assessment: the assessment of performance at work , 2005, Medical education.

[27]  J. Norcini,et al.  Work based assessment , 2003, BMJ : British Medical Journal.

[28]  Jeanette N Cleveland,et al.  Raters who pursue different goals give different ratings. , 2004, The Journal of applied psychology.

[29]  J. Norcini,et al.  Workplace-based assessment as an educational tool: AMEE Guide No. 31 , 2007, Medical teacher.

[30]  Wim H. Gijselaers,et al.  Understanding managerial problem-solving, knowledge use and information processing: Investigating stages from school to the workplace , 2006 .

[31]  Katherine Cushing,et al.  Taking a Second Look: Expert and Novice Differences when Observing the Same Classroom Teaching Segment a Second Time , 2000 .

[32]  Kevin W. Eva,et al.  The Cambridge Handbook of Expertise and Expert Performance: Expertise in Medicine and Surgery , 2006 .

[33]  C. V. D. van der Vleuten,et al.  The assessment of professional competence: building blocks for theory development. , 2010, Best practice & research. Clinical obstetrics & gynaecology.

[34]  Andy P. Field,et al.  Discovering Statistics Using SPSS , 2000 .

[35]  F. Gobet,et al.  The Cambridge handbook of expertise and expert performance , 2006 .

[36]  Angelo S. DeNisi,et al.  A cognitive approach to performance appraisal : a program of research , 1996 .

[37]  L. Sulsky,et al.  Using frame-of-reference training to understand the implications of rater idiosyncrasy for rating accuracy. , 2008, The Journal of applied psychology.

[38]  P. Levy,et al.  The Social Context of Performance Appraisal: A Review and Framework for the Future , 2004 .

[39]  M. London.  How people evaluate others in organizations , 2001 .

[40]  J. Uleman,et al.  Spontaneous inferences, implicit impressions, and implicit theories. , 2008, Annual review of psychology.

[41]  G. Andrews,et al.  Mja Guidelines for Assessing Qualitative Research Quality in Qualitative Research Criteria for Authors and Assessors in the Submission and Assessment of Qualitative Research Articles for the Medical Journal of Australia , 2022 .

[42]  Julian Archer,et al.  Assuring validity of multisource feedback in a national programme , 2010, Archives of Disease in Childhood.

[43]  A. Muijtjens,et al.  Workplace-based assessment: effects of rater expertise , 2010, Advances in health sciences education : theory and practice.

[44]  F. Gobet.  The Cambridge handbook of expertise and expert performance , 2006 .

[45]  Charles E. Lance,et al.  RATER SOURCE EFFECTS ARE ALIVE AND WELL AFTER ALL , 2010 .

[46]  Jonathan Sherbino,et al.  The role of assessment in competency-based medical education , 2010, Medical teacher.

[47]  Walter C. Borman,et al.  Personal constructs, performance schemata, and “folk theories” of subordinate effectiveness: Explorations in an army officer sample☆ , 1987 .

[48]  Gary Klein,et al.  Naturalistic Decision Making , 2008, Hum. Factors.

[49]  Eliot R. Smith,et al.  Situated Social Cognition , 2007 .

[50]  Steven M. Downing,et al.  Assessment in Health Professions Education , 2009 .

[51]  Chockalingam Viswesvaran,et al.  Comparative analysis of the reliability of job performance ratings , 1996 .

[52]  S. Lurie,et al.  Measurement of the General Competencies of the Accreditation Council for Graduate Medical Education: A Systematic Review , 2009, Academic medicine : journal of the Association of American Medical Colleges.

[53]  David R. Thomas,et al.  A General Inductive Approach for Analyzing Qualitative Evaluation Data , 2006 .

[54]  Eric S Holmboe,et al.  Effects of Training in Direct Observation of Medical Residents' Clinical Competence , 2004, Annals of Internal Medicine.

[55]  M A Albanese.  Challenges in using rater judgements in medical education. , 2000, Journal of evaluation in clinical practice.

[56]  Helvi Kyngäs,et al.  The qualitative content analysis process. , 2008, Journal of advanced nursing.

[57]  Rosaline S Barbour,et al.  Checklists for improving rigour in qualitative research: a case of the tail wagging the dog? , 2001, BMJ : British Medical Journal.

[58]  William C McGaghie,et al.  SPECIAL ARTICLE: Cognitive, Social and Environmental Sources of Bias in Clinical Performance Ratings , 2003, Teaching and learning in medicine.