A method for identifying extreme OSCE examiners

Background:  Performance assessments rely on human judgment, and are vulnerable to rater effects (e.g. leniency or harshness). Making valid inferences from performance ratings for high‐stakes decisions requires the management of rater effects. A simple method for detecting extreme raters that does not require sophisticated statistical knowledge or software has been developed as part of the quality assurance process for objective structured clinical examinations (OSCEs). We believe it is applicable to a range of examinations that rely on human raters.

[1]  C. Roberts,et al.  Should candidate scores be adjusted for interviewer stringency or leniency in the multiple mini‐interview? , 2010, Medical education.

[2]  Brian E. Clauser,et al.  An Examination of Rater Drift within a Generalizability Theory Framework. , 2009 .

[3]  Wayne Woloschuk,et al.  Undesired variance due to examiner stringency/leniency effect in communication skill scores assessed in OSCEs , 2008, Advances in health sciences education : theory and practice.

[4]  R. Yudkowsky,et al.  Rater Errors in a Clinical Skills Assessment of Medical Students , 2007, Evaluation & the health professions.

[5]  I. McManus,et al.  Bmc Medical Education Assessment of Examiner Leniency and Stringency ('hawk-dove Effect') in the Mrcp(uk) Clinical Examination (paces) Using Multi-facet Rasch Modelling , 2006 .

[6]  S. Downing.  Threats to the validity of clinical teaching assessments: What about rater error? , 2005, Medical education.

[7]  H. Burstin,et al.  The importance of clinical outcomes in medical education research , 2005, Medical education.

[8]  S. Kaney,et al.  Examiner fatigue in communication skills objective structured clinical examinations , 2001, Medical education.

[9]  Chockalingam Viswesvaran,et al.  Least Squares Models to Correct for Rater Effects in Performance Assessment , 1993 .

[10]  P. Maguire,et al.  Assessing clinical competence. , 1989, BMJ.

[11]  H. John Bernardin,et al.  Strategies in Rater Training , 1981 .

[12]  John M. Ivancevich,et al.  Longitudinal study of the effects of rater training on psychometric error in ratings. , 1979 .

[13]  R. Hambleton,et al.  Quality Assurance Methods for Performance-Based Assessments , 2003, Advances in health sciences education : theory and practice.

[14]  W. Simon.  Oral examinations. , 1947, Northwest dentistry.