Defining and assessing professional competence.

CONTEXT Current assessment formats for physicians and trainees reliably test core knowledge and basic skills. However, they may underemphasize some important domains of professional medical practice, including interpersonal skills, lifelong learning, professionalism, and integration of core knowledge into clinical practice. OBJECTIVES To propose a definition of professional competence, to review current means for assessing it, and to suggest new approaches to assessment. DATA SOURCES We searched the MEDLINE database from 1966 to 2001 and reference lists of relevant articles for English-language studies of reliability or validity of measures of competence of physicians, medical students, and residents. STUDY SELECTION We excluded articles of a purely descriptive nature, duplicate reports, reviews, and opinions and position statements, which yielded 195 relevant citations. DATA EXTRACTION Data were abstracted by 1 of us (R.M.E.). Quality criteria for inclusion were broad, given the heterogeneity of interventions, complexity of outcome measures, and paucity of randomized or longitudinal study designs. DATA SYNTHESIS We generated an inclusive definition of competence: the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and the community being served. Aside from protecting the public and limiting access to advanced training, assessments should foster habits of learning and self-reflection and drive institutional change. Subjective, multiple-choice, and standardized patient assessments, although reliable, underemphasize important domains of professional competence: integration of knowledge and skills, context of care, information management, teamwork, health systems, and patient-physician relationships. Few assessments observe trainees in real-life situations, incorporate the perspectives of peers and patients, or use measures that predict clinical outcomes. CONCLUSIONS In addition to assessments of basic skills, new formats that assess clinical reasoning, expert judgment, management of ambiguity, professionalism, time management, learning strategies, and teamwork promise a multidimensional assessment while maintaining adequate reliability and validity. Institutional support, reflection, and mentoring must accompany the development of assessment programs.

[1]  D. Saltzberg,et al.  Reinforcement of self‐directed learning and the development of professional attitudes through peer‐ and self‐assessment , 1998, Academic medicine : journal of the Association of American Medical Colleges.

[2]  Victoria Hoban,et al.  The Reflective Practitioner , 2013 .

[3]  R L Helmreich,et al.  Virtual reality in medical training. , 1997, CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne.

[4]  R L Helmreich,et al.  Anaesthetic simulation and lessons to be learned from aviation , 1997, Canadian journal of anaesthesia = Journal canadien d'anesthesie.

[5]  J. Rethans,et al.  Assessment in general practice: the predictive value of written‐knowledge tests and a multiple‐station examination for actual medical performance in daily practice , 1999, Medical education.

[6]  C. V. D. van der Vleuten,et al.  Standardized or real patients to test clinical competence? The long case revisited , 2001, Medical education.

[7]  D. Goleman Emotional Intelligence. New York (Bantam) 1995. , 1995 .

[8]  G. Webster,et al.  A Comparison of Knowledge, Synthesis, and Clinical Judgment , 1984, Evaluation & the health professions.

[9]  H. Barrows,et al.  Six years of comprehensive, clinical, performance‐based assessment using standardized patients at the Southern Illinois University School of Medicine , 1992, Academic medicine : journal of the Association of American Medical Colleges.

[10]  A. Dietrich,et al.  The Periodic Health Examination Provided to Asymptomatic Older Women: An Assessment Using Standardized Patients , 1993, Annals of Internal Medicine.

[11]  Validity of final examinations in undergraduate medical training , 2000, BMJ : British Medical Journal.

[12]  J. Rethans,et al.  Assessment of general practitioners by video observation of communicative and medical performance in daily practice: issues of validity, reliability and feasibility , 1999, Medical education.

[13]  J. Ely,et al.  Malpractice claims against family physicians are the best doctors sued more? , 1999, The Journal of family practice.

[14]  J. Kleijnen,et al.  Influence of context effects on health outcomes: a systematic review , 2001, The Lancet.

[15]  G Bordage,et al.  Elaborated knowledge: a key to successful diagnostic thinking , 1994, Academic medicine : journal of the Association of American Medical Colleges.

[16]  E. Mumford,et al.  Ratings of videotaped simulated patient interviews and four other methods of evaluating a psychiatry clerkship. , 1987, The American journal of psychiatry.

[17]  R. Reznick,et al.  A new assessment tool: the patient assessment and management examination. , 1997, Surgery.

[18]  H. Schmidt,et al.  A Cognitive Perspective on Medical Expertise: Theory and Implications , 1990 .

[19]  D M Irby,et al.  Teaching and learning in ambulatory care settings: a thematic review of the literature , 1995, Academic medicine : journal of the Association of American Medical Colleges.

[20]  K. Fiscella,et al.  Influence of Patient Education on Profiles of Physician Practices , 1999, Annals of Internal Medicine.

[21]  C. Tresolini Task Force on Advancing Psychosocial Health Education , 1994 .

[22]  J Norcini,et al.  Association between licensing examination scores and resource use and quality of care in primary care practice. , 1998, JAMA.

[23]  Robyn Tamblyn,et al.  Can standardized patients predict real‐patient satisfaction with the doctor‐patient relationship? , 1994 .

[24]  G J Holsgrove,et al.  Guide to postgraduate exams: multiple-choice questions. , 1992, British journal of hospital medicine.

[25]  D. Klass,et al.  Setting examination-level standards for a performance-based assessment of physicians' clinical skills. , 1998, Academic medicine : journal of the Association of American Medical Colleges.

[26]  R. Reznick,et al.  Who should rate candidates in an objective structured clinical examination? , 1996, Academic medicine : journal of the Association of American Medical Colleges.

[27]  G. Miller The assessment of clinical skills/competence/performance , 1990, Academic medicine : journal of the Association of American Medical Colleges.

[28]  C. V. D. van der Vleuten,et al.  Formative assessment of the consultation performance of medical students in the setting of general practice using a modified version of the Leicester Assessment Package , 2000, Medical Education.

[29]  A. Dietrich,et al.  Recognizing and managing depression in primary care: a standardized patient study. , 1999, The Journal of family practice.

[30]  Alastair Baker,et al.  Crossing the Quality Chasm: A New Health System for the 21st Century , 2001, BMJ : British Medical Journal.

[31]  R. M. Smith,et al.  The triple‐jump examination as an assessment tool in the problem‐based medical curriculum at the University of Hawaii , 1993, Academic medicine : journal of the Association of American Medical Colleges.

[32]  R. Helmreich,et al.  The importance of human factors in the operating room. , 1994, Anesthesiology.

[33]  D M Kaufman,et al.  A Comparison of Standard‐setting Procedures for an OSCE in Undergraduate Medical Education , 2000, Academic medicine : journal of the Association of American Medical Colleges.

[34]  R. Gross,et al.  The influence of patient-practitioner agreement on outcome of care. , 1981, American journal of public health.

[35]  J. Colliver,et al.  Validating the standardized‐patient assessment administered to medical students in the New York City Consortium , 1997, Academic medicine : journal of the Association of American Medical Colleges.

[36]  J. Macnaughton,et al.  Clinical Judgement: Evidence in Practice , 2000 .

[37]  A. Laduca,et al.  Validation of Professional Licensure Examinations , 1994, Evaluation & the health professions.

[38]  R. Epstein,et al.  Toward creating physician-healers: fostering medical students' self-awareness, personal growth, and well-being. , 1999, Academic medicine : journal of the Association of American Medical Colleges.

[39]  R. Epstein,et al.  Mindful practice. , 1999, JAMA.

[40]  Holsgrove Gj,et al.  Guide to postgraduate exams: multiple-choice questions. , 1992 .

[41]  M. Stewart Effective physician-patient communication and health outcomes: a review. , 1995, CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne.

[42]  G. Norman,et al.  Pitfalls in the pursuit of objectivity: issues of reliability , 1991, Medical education.

[43]  G Bordage,et al.  The structure of medical knowledge in the memories of medical students and general practitioners: categories and prototypes , 1984, Medical education.

[44]  D. Swanson,et al.  Validity of NBME Part I and Part II scores for selection of residents in orthopaedic surgery, dermatology, and preventive medicine , 1993, Academic medicine : journal of the Association of American Medical Colleges.

[45]  R. Tamblyn,et al.  Factors associated with the accuracy of standardized patient presentation , 1990, Academic medicine : journal of the Association of American Medical Colleges.

[46]  P Glassman,et al.  How well does chart abstraction measure quality? A prospective comparison of standardized patients with the medical record. , 2000, The American journal of medicine.

[47]  G. Dunnington,et al.  A pilot experience with competency-based clinical skills assessment in a surgical clerkship. , 1994, American journal of surgery.

[48]  P. Grand'maison,et al.  Large-scale use of an objective, structured clinical examination for licensing family physicians. , 1992, CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne.

[49]  J. Sexton,et al.  Error, stress, and teamwork in medicine and aviation: cross sectional surveys , 2000, BMJ : British Medical Journal.

[50]  J. Colliver,et al.  Assessing clinical performance with standardized patients. , 1997, JAMA.

[51]  R. Joynt Descartes' Error: Emotion, Reason, and the Human Brain , 1995 .

[52]  Andrew Gonczi The development of competency-based assessment strategies for the professions , 1993 .

[53]  J. Sinacore,et al.  Medical student errors in making a diagnosis. , 1998, Academic medicine : journal of the Association of American Medical Colleges.

[54]  T. Arbizu,et al.  Clinical skills assessment with standardized patients , 1997, Medical education.

[55]  D. Kassebaum,et al.  Shortcomings in the evaluation of students' clinical skills and behaviors in medical school. , 1999, Academic medicine : journal of the Association of American Medical Colleges.

[56]  D. Klass,et al.  Standardized patients' accuracy in recording examinees' behaviors using checklists , 1997, Academic medicine : journal of the Association of American Medical Colleges.

[57]  R. Helmreich,et al.  Surgery as team endeavour , 2000, Canadian journal of anaesthesia = Journal canadien d'anesthesie.

[58]  Reed G. Williams,et al.  Scoring and Standard Setting with Standardized Patients , 1993 .

[59]  A. Laduca,et al.  The Design of a New Physician Licensure Examination , 1984, Evaluation & the health professions.

[60]  C. V. D. van der Vleuten,et al.  Composite undergraduate clinical examinations: how should the components be combined to maximize reliability? , 2001, Medical education.

[61]  P. Maurette,et al.  [To err is human: building a safer health system]. , 2002, Annales francaises d'anesthesie et de reanimation.

[62]  M. Stewart,et al.  The impact of patient-centered care on outcomes. , 2000, The Journal of family practice.

[63]  R. Tamblyn,et al.  The accuracy of standardized patient presentation , 1991, Medical education.

[64]  Richard E. Mayer,et al.  From Novice to expert , 1988 .

[65]  N. Coe,et al.  Influence of effective communication by surgery students on their oral examination scores , 1991, Academic medicine : journal of the Association of American Medical Colleges.

[66]  J. Colliver,et al.  Global ratings of videotaped performance versus global ratings of actions recorded on checklists: a criterion for performance assessment with standardized patients. , 1999, Academic medicine : journal of the Association of American Medical Colleges.

[67]  B. Ferrell,et al.  Standardized patients: a long‐station clinical examination format , 1993, Medical education.

[68]  E. Petrusa,et al.  Reliability and validity of an objective structured clinical examination for assessing the clinical performance of residents. , 1990, Archives of internal medicine.

[69]  L. Candib Medicine And The Family: A Feminist Perspective , 1995 .

[70]  S. Kaplan,et al.  Assessing the Effects of Physician-Patient Interactions on the Outcomes of Chronic Disease , 1989, Medical care.

[71]  G. Dinant,et al.  Do short cases elicit different thinking processes than factual knowledge questions do? , 2001, Medical education.

[72]  D. Swanson,et al.  The relationship between clinical science performance in 20 medical schools and performance on Step 2 of the USMLE licensing examination. 1994‐95 Validity Study Group for USMLE Step 1 and 2 Pass/Fail Standards , 1996, Academic medicine : journal of the Association of American Medical Colleges.

[73]  J. Dewey Experience and Nature , 1960 .

[74]  Michael E. Kane Model-Based Practice Analysis and Test Specifications , 1997 .

[75]  R. Epstein,et al.  Calibrating the Physician: Personal Awareness and Effective Patient Care , 1997 .

[76]  J. Spierdijk,et al.  Reported significant observations during anaesthesia: a prospective analysis over an 18-month period. , 1992, British journal of anaesthesia.

[77]  J. Rethans,et al.  Assessment of practicing family physicians: comparison of observation in a multiple-station examination using standardized patients with observation of consultations in daily practice. , 1999, Academic medicine : journal of the Association of American Medical Colleges.

[78]  R. Stillman,et al.  Evaluation of the student: improving validity of the oral examination. , 1983, Surgery.

[79]  S. Kaplan,et al.  Expanding patient involvement in care. Effects on patient outcomes. , 1985, Annals of internal medicine.

[80]  D. Swanson,et al.  Relationship between scores on NBME basic science subject tests and the first administration of the newly designed NBME Part I examination , 1992, Academic medicine : journal of the Association of American Medical Colleges.

[81]  R. Hiscock,et al.  Predictors of Outcome in Headache Patients Presenting to Family Physicians ‐ a One Year Prospective Study , 1986, Headache.

[82]  G. Regehr,et al.  OSCE checklists do not capture increasing levels of expertise. , 1999, Academic medicine : journal of the Association of American Medical Colleges.

[83]  Dauphinee Wd Assessing clinical performance. Where do we stand and what might we expect , 1995 .

[84]  T. Hassard,et al.  Assessing practicing physicians in two settings using standardized patients , 1992, Academic medicine : journal of the Association of American Medical Colleges.

[85]  P. Tutton Psychometric test results associated with high achievement in basic science components of a medical curriculum , 1996, Academic medicine : journal of the Association of American Medical Colleges.

[86]  J. Carline,et al.  Use of peer ratings to evaluate physician performance. , 1993, JAMA.

[87]  David B. Swanson,et al.  Assessment of clinical skills with standardized patients: State of the art , 1990 .

[88]  D. Safran,et al.  Switching doctors: predictors of voluntary disenrollment from a primary physician's practice. , 2001, The Journal of family practice.

[89]  J. Colliver,et al.  Clinical performance‐based test sensitivity and specificity in predicting first‐year residency performance , 1993, Academic medicine : journal of the Association of American Medical Colleges.

[90]  G. Barnas,et al.  Effect of student and preceptor gender on clinical grades in an ambulatory care clerkship , 1995, Academic medicine : journal of the Association of American Medical Colleges.

[91]  A. A-Latif An examination of the examinations: the reliability of the objective structured clinical examination and clinical examination. , 1992, Medical teacher.

[92]  Alvan Feinstein,et al.  Clinical Judgment Revisited: The Distraction of Quantitative Models , 1994, Annals of Internal Medicine.

[93]  W. Dauphinee Assessing clinical performance. Where do we stand and what might we expect? , 1995, JAMA.

[94]  D. Schoen The Reflective Practitioner , 1983 .

[95]  M. Stewart,et al.  Evidence on patient-doctor communication. , 1999, Cancer prevention & control : CPC = Prevention & controle en cancerologie : PCC.

[96]  Cees van der Vleuten,et al.  Developing Case‐specific Checklists for Standardized‐patient—Based Assessments in Internal Medicine: A Review of the Literature , 2000, Academic medicine : journal of the Association of American Medical Colleges.

[97]  J. Woolliscroft,et al.  Assessment of a pattern‐recognition examination in a clinical clerkship , 1994, Academic medicine : journal of the Association of American Medical Colleges.

[98]  M. Philbin,et al.  Assessment of clinical skills of residents utilizing standardized patients. A follow-up study and recommendations for application. , 1991, Annals of internal medicine.

[99]  L. Ross,et al.  An expert‐judgment approach to setting standards for a standardized‐patient examination , 1996, Academic medicine : journal of the Association of American Medical Colleges.

[100]  C. Blane,et al.  Objectively evaluating student case presentations. , 1984, Investigative radiology.

[101]  A. Dietrich,et al.  A standardized‐patient assessment of a continuing medical education program to improve physicians' cancer‐control clinical skills , 1995, Academic medicine : journal of the Association of American Medical Colleges.

[102]  M. Abrahamowicz,et al.  First‐visit bias in the measurement of clinical competence with standardized patients , 1992, Academic medicine : journal of the Association of American Medical Colleges.

[103]  G. Norman,et al.  Performance‐based assessment of clinical ethics using an objective structured clinical examination , 1996, Academic medicine : journal of the Association of American Medical Colleges.

[104]  R. Shaw Democracy and Excellence , 1988 .

[105]  M. Cusimano,et al.  Defining standards of competent performance on an OSCE. , 1998, Academic medicine : journal of the Association of American Medical Colleges.

[106]  J. Benbassat,et al.  Reliability of global rating scales in the assessment of clinical competence of medical students , 1987, Medical education.

[107]  Gordon Page,et al.  Guidelines for assessing clinical competence , 1994 .

[108]  A. Rothman,et al.  Validity and Generalizability of Global Ratings in an Objective Structured Clinical Examination , 1991, Academic medicine : journal of the Association of American Medical Colleges.

[109]  G. Webster,et al.  Reliability, validity and efficiency of multiple choice question and patient management problem item formats in assessment of clinical competence , 1985, Medical education.

[110]  C. Violato,et al.  Feasibility and psychometric properties of using peers, consulting physicians, co‐workers, and patients to assess physicians , 1997, Academic medicine : journal of the Association of American Medical Colleges.

[111]  R. Frankel,et al.  The doctor-patient relationship and malpractice. Lessons from plaintiff depositions. , 1994, Archives of internal medicine.

[112]  J. R. Burnett,et al.  Democracy and Excellence in american secondary Education. , 1964 .

[113]  E. Emanuel,et al.  Preserving the physician-patient relationship in the era of managed care. , 1995, JAMA.

[114]  M. Eraut Learning Professional Processes: Public Knowledge and Personal Experience , 2002 .

[115]  M. J. Gordon A review of the validity and accuracy of self‐assessments in health professions training , 1991, Academic medicine : journal of the Association of American Medical Colleges.

[116]  D. Klass Reevaluation of clinical competency. , 2000, American journal of physical medicine & rehabilitation.

[117]  S. Satya‐Murti Evidence-based Medicine: How to Practice and Teach EBM , 1997 .

[118]  P. H. Walker,et al.  Partnerships in healthcare : transforming relational process , 1998 .

[119]  S. Downing The assessment of clinical competence on the emergency medicine specialty certification examination: the validity of clinically relevant multiple-choice items. , 1980, Annals of emergency medicine.

[120]  R S Newbower,et al.  Preventable anesthesia mishaps: a study of human factors* , 1978, Anesthesiology.

[121]  D. Matsell,et al.  Reliability and validity of the objective structured clinical examination in paediatrics , 1991, Medical education.

[122]  J. Bauer,et al.  Checklist self-evaluation in a standardized patient exercise. , 1995, American journal of surgery.

[123]  R. Epstein,et al.  Communication between primary care physicians and consultants. , 1995, Archives of family medicine.

[124]  R. Helmreich On error management: lessons from aviation , 2000, BMJ : British Medical Journal.

[125]  J. Ramsay,et al.  Detecting and correcting for rater‐induced differences in standardized patient tests of clinical competence , 1990, Academic medicine : journal of the Association of American Medical Colleges.

[126]  D. Newble,et al.  Psychometric characteristics of the objective structured clinical examination , 1988, Medical education.

[127]  Judith A. Hall,et al.  Improving physicians' interviewing skills and reducing patients' emotional distress. A randomized clinical trial. , 1995, Archives of internal medicine.

[128]  R Tamblyn,et al.  Unnecessary Prescribing of NSAIDs and the Management of NSAID-Related Gastropathy in Medical Practice , 1997, Annals of Internal Medicine.

[129]  T. Konrad,et al.  Responsive medical professionalism: integrating education, practice, and community in a market‐driven era , 1998, Academic medicine : journal of the Association of American Medical Colleges.

[130]  M. Polanyi,et al.  Personal Knowledge: Towards a post-critical philosophy , 1959 .

[131]  E. Hundert,et al.  Characteristics of the informal curriculum and trainees' ethical choices. , 1996, Academic medicine : journal of the Association of American Medical Colleges.

[132]  P. Tate,et al.  Assessing physicians' interpersonal skills via videotaped encounters: a new approach for the Royal College of General Practitioners Membership examination. , 1999, Journal of health communication.

[133]  C. V. D. van der Vleuten,et al.  The Script Concordance Test: A Tool to Assess the Reflective Clinician , 2000, Teaching and learning in medicine.

[134]  C. V. D. van der Vleuten,et al.  Measurement of clinical reflective capacity early in training as a predictor of clinical reasoning performance at the end of residency: an experimental study on the script concordance test , 2001, Medical education.

[135]  R. Tamblyn,et al.  Use of standardized patients in the assessment of medical practice. , 1998, CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne.

[136]  G. Dunnington,et al.  Structured single-observer methods of evaluation for the assessment of ward performance on the surgical clerkship. , 1990, American journal of surgery.

[137]  P Marsden,et al.  Clinical problem solving: the beginning of the process , 1982, Medical education.

[138]  J. Ware,et al.  Linking primary care performance to outcomes of care. , 1998, The Journal of family practice.

[139]  H. Barrows An overview of the uses of standardized patients for teaching and evaluating clinical skills. AAMC , 1993, Academic medicine : journal of the Association of American Medical Colleges.

[140]  J. Horder THE CARE OF THE PATIENT. , 1963, Medical world.

[141]  R. A. Melker,et al.  Simulated patients in assessing consultation skills of trainees in general practice vocational training: a validity study , 1994, Medical education.

[142]  J. Norcini,et al.  The Mini-CEX (Clinical Evaluation Exercise): A Preliminary Investigation , 1995, Annals of Internal Medicine.

[143]  B. Joorabchi Objective structured clinical examination in a pediatric residency program. , 1991, American journal of diseases of children.

[144]  Ronald Epstein,et al.  Time and the patient-physician relationship , 1999, Journal of General Internal Medicine.

[145]  D. Laube,et al.  Improvement of reliability of an oral examination by a structured evaluation instrument. , 1983, Journal of medical education.

[146]  Manuel London,et al.  Maximizing the Value of 360-degree Feedback: A Process for Successful Individual and Organizational Development , 1998 .

[147]  R W Sanson-Fisher,et al.  Simulated patients and the assessment of medical students' interpersonal skills , 1980, Medical education.

[148]  A. Scherpbier,et al.  The effect on reliability of adding a separate written assessment component to an objective structured clinical examination , 2000, Medical education.

[149]  A. Damasio Descartes' error: emotion, reason, and the human brain. avon books , 1994 .

[150]  Dave Davis,et al.  Does CME Work? An Analysis of the Effect of Educational Activities on Physician Performance or Health Care Outcomes , 1998, International journal of psychiatry in medicine.

[151]  F. Vella,et al.  The power of mindful learning , 1997 .

[152]  P. Harasym,et al.  Helping students learn to think like experts when solving clinical problems , 1997, Academic medicine : journal of the Association of American Medical Colleges.

[153]  J J Veloski,et al.  Patients don't present with five choices: an alternative to multiple-choice tests in assessing physicians' competence. , 1999, Academic medicine : journal of the Association of American Medical Colleges.

[154]  M. Polanyi Personal Knowledge: Towards a post-critical philosophy , 1959 .

[155]  M. Polanyi The Logic of Tacit Inference , 1966, Philosophy.

[156]  H. Barrows,et al.  Validity of simple approach to scoring and standard setting for standardized‐patient cases in an examination of clinical competence , 1996, Academic medicine : journal of the Association of American Medical Colleges.

[157]  M. Abrahamowicz,et al.  Use of standardized patients to assess between-physician variations in resource utilization. , 1997, JAMA.