Broadening Perspectives on Clinical Performance Assessment: Rethinking the Nature of In-training Assessment

ContextIn-training assessment (ITA), defined as multiple assessments of performance in the setting of day-to-day practice, is an invaluable tool in assessment programmes which aim to assess professional competence in a comprehensive and valid way. Research on clinical performance ratings, however, consistently shows weaknesses concerning accuracy, reliability and validity. Attempts to improve the psychometric characteristics of ITA focusing on standardisation and objectivity of measurement thus far result in limited improvement of ITA-practices.PurposeThe aim of the paper is to demonstrate that the psychometric framework may limit more meaningful educational approaches to performance assessment, because it does not take into account key issues in the mechanics of the assessment process. Based on insights from other disciplines, we propose an approach to ITA that takes a constructivist, social-psychological perspective and integrates elements of theories of cognition, motivation and decision making. A central assumption in the proposed framework is that performance assessment is a judgment and decision making process, in which rating outcomes are influenced by interactions between individuals and the social context in which assessment occurs.DiscussionThe issues raised in the article and the proposed assessment framework bring forward a number of implications for current performance assessment practice. It is argued that focusing on the context of performance assessment may be more effective in improving ITA practices than focusing strictly on raters and rating instruments. Furthermore, the constructivist approach towards assessment has important implications for assessment procedures as well as the evaluation of assessment quality. Finally, it is argued that further research into performance assessment should contribute towards a better understanding of the factors that influence rating outcomes, such as rater motivation, assessment procedures and other contextual variables.

[1]  M. Chi,et al.  The Nature of Expertise , 1988 .

[2]  C. van Barneveld The Dependability of Medical Students’ Performance Ratings as Documented on In-Training Evaluations , 2005, Academic medicine : journal of the Association of American Medical Colleges.

[3]  S. J. Motowidlo,et al.  Effects of Accountability on Rating Behavior and Rater Accuracy , 2003 .

[4]  Eleanor Hawe,et al.  'It's Pretty Difficult to Fail': The reluctance of lecturers to award a failing grade , 2003 .

[5]  Lorne M. Sulsky,et al.  Performance appraisal in the changing world of work: Implications for the meaning and measurement of work performance. , 1998 .

[6]  Norbert Schwarz,et al.  Constructing Perceptions of Vulnerability: Personal Relevance and the Use of Experiential Information in Health Judgments , 1998 .

[7]  E. Guba,et al.  Fourth Generation Evaluation , 1989 .

[8]  Organization of information in memory and the performance appraisal process: evidence from the field. , 1996 .

[9]  S. Zedeck A process analysis of the assessment center method. , 1986 .

[10]  Cees P. M. van der Vleuten,et al.  Assessing professional competence: from methods to programmes , 2005 .

[11]  G. G. Nahum Evaluating medical student obstetrics and gynecology clerkship performance: which assessment tools are most reliable? , 2004, American journal of obstetrics and gynecology.

[12]  Thomas P. Cafferty,et al.  Organization of information used for performance appraisals: role of diary-keeping , 1989 .

[13]  R. Reisenzein,et al.  Effects of Mood on Evaluative Judgements: Influence of Reduced Processing Capacity and Mood Salience , 1998 .

[14]  Chris Rust,et al.  A social constructivist assessment process model: how the research literature shows us this could be best practice , 2005 .

[15]  D. Solomon,et al.  Grade Inflation in Internal Medicine Clerkships: Results of a National Survey , 2000, Teaching and learning in medicine.

[16]  Philip L. Smith,et al.  Contextualizing the Interpretation of Reliability Data , 1998 .

[17]  Philip E. Tetlock,et al.  Accountability and complexity of thought. , 1983 .

[18]  R. Cardy,et al.  The effects of individual performance schemata and dimension familiarization on rating accuracy , 1987 .

[19]  Richard J. Klimoski,et al.  Accountability forces in performance appraisal , 1990 .

[20]  Eric S. Holmboe,et al.  Faculty and the Observation of Trainees’ Clinical Skills: Problems and Opportunities , 2004, Academic medicine : journal of the Association of American Medical Colleges.

[21]  F. Lievens,et al.  Assessor training strategies and their effects on accuracy, interrater reliability, and discriminant validity. , 2001, The Journal of applied psychology.

[22]  Walter C. Borman,et al.  Task Performance and Contextual Performance: The Meaning for Personnel Selection Research , 1997 .

[23]  S. J. Motowidlo,et al.  Effects of Rater Accountability on the Accuracy and the Favorability of Performance Ratings , 2004 .

[24]  Eileen Piggot‐Irvine,et al.  Key features of appraisal effectiveness , 2003 .

[25]  Ginette Delandshere,et al.  Assessment of Complex Performances: Limitations of Key Measurement Assumptions , 1998 .

[26]  William K. Balzer,et al.  Rater errors and rating accuracy. , 1989 .

[27]  C. V. D. van der Vleuten,et al.  The assessment of professional competence: Developments, research and practical implications. , 1996, Advances in health sciences education : theory and practice.

[28]  Diana H. J. M. Dolmans,et al.  International handbook of research in medical education , 2002 .

[29]  G. Eiger,et al.  Do Global Rating Forms Enable Program Directors to Assess the ACGME Competencies? , 2004, Academic medicine : journal of the Association of American Medical Colleges.

[30]  Jeff W. Johnson,et al.  The relative importance of task and contextual performance dimensions to supervisor judgments of overall performance. , 2001, The Journal of applied psychology.

[31]  J. Forgas Feeling and Doing: Affective Influences on Interpersonal Behavior , 2002 .

[32]  J. Norcini,et al.  Facing the challenges of competency‐based assessment of postgraduate dental training: Longitudinal Evaluation of Performance (LEP) , 2002, Medical education.

[33]  Diana H. J. M. Dolmans,et al.  Quality issues in judging portfolios: implications for organizing teaching portfolio assessment procedures , 2005 .

[34]  William K. Balzer,et al.  Systematic distortions in memory-based behavior ratings and performance evaluations: Consequences for rating accuracy. , 1986 .

[35]  Brenda Johnston,et al.  Summative assessment of portfolios: an examination of different approaches to agreement over outcomes , 2004 .

[36]  Kevin R. Murphy,et al.  Multiple uses of performance appraisal: Prevalence and correlates. , 1989 .

[37]  P H Harasym,et al.  Diagnostic reasoning strategies and diagnostic success , 2003, Medical education.

[38]  Michael A. Hogg,et al.  Introducing social psychology , 2003 .

[39]  M. Taylor,et al.  Due Process in Performance Appraisal: A Quasi-Experiment in Procedural Justice , 1995 .

[40]  T. Crooks The Impact of Classroom Evaluation Practices on Students , 1988 .

[41]  Juan I. Sanchez,et al.  A second look at the relationship between rating and behavioral accuracy in performance appraisal. , 1996 .

[42]  Joseph P. Forgas,et al.  Affective Influences on Judgments and Behavior in Organizations: An Information Processing Perspective , 2001 .

[43]  J. Donaldson,et al.  Contextual tensions of the clinical environment and their influence on teaching and learning , 2004, Medical education.

[44]  S. Hodder,et al.  Validity of three clinical performance assessments of internal medicine clerks , 1995, Academic medicine : journal of the Association of American Medical Colleges.

[45]  G. Regehr,et al.  The Effect of Candidates' Perceptions of the Evaluation Method on Reliability of Checklist and Global Rating Scores in an Objective Structured Clinical Examination , 2002, Academic medicine : journal of the Association of American Medical Colleges.

[46]  T. Macan,et al.  Note-taking in the employment interview: effects on recall and judgments. , 2002, The Journal of applied psychology.

[47]  R. Goffin,et al.  Can performance-feedback accuracy be improved? Effects of rater priming and rating-scale format on rating accuracy. , 2001, The Journal of applied psychology.

[48]  L. Pangaro,et al.  How well do internal medicine faculty members evaluate the clinical skills of residents? , 1992, Annals of internal medicine.

[49]  C. Barneveld,et al.  Assessment of Clinical Performance: In-Training Evaluation , 2002 .

[50]  L. Pangaro Investing in descriptive evaluation: a vision for the future of assessment , 2000, Medical teacher.

[51]  Deidra J. Schleicher,et al.  A field study of the effects of rating purpose on the quality of multisource ratings. , 2003 .

[52]  M. Donnelly,et al.  Faculty sensitivity in detecting medical students' clinical competence , 1995 .

[53]  C VanBarneveld,et al.  The dependability of medical students' performance ratings as documented on in-training evaluations. , 2005 .

[54]  G. Norman Research in clinical reasoning: past history and current trends , 2005, Medical education.

[55]  Deidra J. Schleicher,et al.  A Cognitive Evaluation of Frame-of-Reference Rater Training: Content and Process Issues. , 1998, Organizational behavior and human decision processes.

[56]  Brian E. Clauser,et al.  The Use of Computers in Assessment , 2002 .

[57]  Gerald R. Ferris,et al.  Social Context of Performance Evaluation Decisions , 1993 .

[58]  L. Komatsu Recent views of conceptual structure , 1992 .

[59]  M. Kahn,et al.  Residency Program Director Evaluations Do Not Correlate With Performance on a Required 4th-Year Objective Structured Clinical Examination , 2001, Teaching and learning in medicine.

[60]  H. Schmidt,et al.  A cognitive perspective on medical expertise: theory and implication [published erratum appears in Acad Med 1992 Apr;67(4):287] , 1990, Academic medicine : journal of the Association of American Medical Colleges.

[61]  P. Wolfson,et al.  Accuracy of surgery clerkship performance raters. , 1991, Academic medicine : journal of the Association of American Medical Colleges.

[62]  Berrin Erdogan,et al.  Procedural Justice as a Two-Dimensional Construct , 2001 .

[63]  Michael M. Harris Rater Motivation in the Performance Appraisal Context: A Theoretical Framework , 1994 .

[64]  Gregory J. Cizek,et al.  Setting performance standards : concepts, methods, and perspectives , 2001 .

[65]  Paul G. Ramsey,et al.  Use of Peer Ratings to Evaluate Physician Performance , 1993 .

[66]  P. Tetlock Accountability: The neglected social context of judgment and choice. , 1985 .

[67]  M B Donnelly,et al.  Ward evaluations: should they be abandoned? , 1997, The Journal of surgical research.

[68]  L. Curry,et al.  Challenge and Vision for Professional Schools in Higher Education@@@Educating Professionals: Responding to New Expectations for Competence and Accountability , 1994 .

[69]  L. Krefting Rigor in qualitative research: the assessment of trustworthiness. , 1991, The American journal of occupational therapy : official publication of the American Occupational Therapy Association.

[70]  J van Tartwijk,et al.  The use of qualitative research criteria for portfolio assessment as an alternative to reliability evaluation: a case study , 2005, Medical education.

[71]  Reed G. Williams,et al.  Do Individual Attendings’ Post-rotation Performance Ratings Detect Residents’ Clinical Performance Deficiencies? , 2004, Academic medicine : journal of the Association of American Medical Colleges.

[72]  David J. Woehr,et al.  Rater training for performance appraisal: A quantitative review , 1994 .

[73]  Liz McDowell,et al.  The Impact of Innovative Assessment on Student Learning , 1995 .

[74]  Charles E. Lance,et al.  Specification of the criterion construct space: An application of hierarchical confirmatory factor analysis. , 1992 .

[75]  J. A. Orban,et al.  Performance Rating as a Function of Trust in Appraisal and Rater Individual Differences. , 1981 .

[76]  Guillermo Solano-Flores,et al.  Performance-Based Assessments , 1994 .

[77]  E. Petrusa Clinical Performance Assessments , 2002 .

[78]  A. Scherpbier,et al.  Clerkship assessment assessed. , 2000, Medical teacher.

[79]  J. Walsh Managerial and Organizational Cognition: Notes from a Trip Down Memory Lane , 1995 .

[80]  J D Gray,et al.  Global rating scales in residency education , 1996, Academic medicine : journal of the Association of American Medical Colleges.

[81]  Kevin R. Murphy,et al.  Understanding Performance Appraisal: Social, Organizational, and Goal-Based Perspectives , 1995 .

[82]  K. Murphy,et al.  Effects of the purpose of rating on accuracy in observing teacher behavior and evaluating teaching performance. , 1984 .

[83]  C. Vleuten,et al.  The Use of Observational Diaries in In-Training Evaluation: Student Perceptions , 2005, Advances in health sciences education : theory and practice.

[84]  James L. Farr,et al.  1 Performance Rating , 2007 .

[85]  William C McGaghie,et al.  SPECIAL ARTICLE: Cognitive, Social and Environmental Sources of Bias in Clinical Performance Ratings , 2003, Teaching and learning in medicine.

[86]  K. Eva What every teacher needs to know about clinical reasoning , 2005, Medical education.

[87]  Angelo S. DeNisi,et al.  The role of appraisal purpose: effects of purpose on information acquisition and utilization , 1985 .

[88]  Shelley E. Taylor,et al.  Social cognition, 2nd ed. , 1991 .

[89]  J. Colliver,et al.  A factor analysis study of performance of first-year residents. , 1986, Journal of medical education.

[90]  Gerard P. Hodgkinson,et al.  The interface of cognitive and industrial, work and organizational psychology , 2003 .

[91]  Charles E. Lance,et al.  A Test of the Context Dependency of Three Causal Models of Halo Rater Error , 1994 .

[92]  Steve W. J. Kozlowski,et al.  The Nature of Conceptual Similarity Schemata: Examination of Some Basic Assumptions , 1992 .

[93]  Jeanette N Cleveland,et al.  Raters who pursue different goals give different ratings. , 2004, The Journal of applied psychology.

[94]  Neil M. A. Hauenstein An information-processing approach to leniency in performance judgments , 1992 .

[95]  S. Messick The Interplay of Evidence and Consequences in the Validation of Performance Assessments , 1994 .