Towards a program of assessment for health professionals: from training into practice

Despite multifaceted attempts to “protect the public,” including the implementation of various assessment practices designed to identify individuals at all stages of training and practice who underperform, profound deficiencies in quality and safety continue to plague the healthcare system. The purpose of this reflections paper is to cast a critical lens on current assessment practices and to offer insights into ways in which they might be adapted to ensure alignment with modern conceptions of health professional education for the ultimate goal of improved healthcare. Three dominant themes will be addressed: (1) The need to redress unintended consequences of competency-based assessment; (2) The potential to design assessment systems that facilitate performance improvement; and (3) The importance of ensuring authentic linkage between assessment and practice. Several principles cut across each of these themes and represent the foundational goals we would put forward as signposts for decision making about the continued evolution of assessment practices in the health professions: (1) Increasing opportunities to promote learning rather than simply measuring performance; (2) Enabling integration across stages of training and practice; and (3) Reinforcing point-in-time assessments with continuous professional development in a way that enhances shared responsibility and accountability between practitioners, educational programs, and testing organizations. Many of the ideas generated represent suggestions for strategies to pilot test, for infrastructure to build, and for harmonization across groups to be enabled. These include novel strategies for OSCE station development, formative (diagnostic) assessment protocols tailored to shed light on the practices of individual clinicians, the use of continuous workplace-based assessment, and broadening the focus of high-stakes decision making beyond determining who passes and who fails. We conclude with reflections on systemic (i.e., cultural) barriers that may need to be overcome to move towards a more integrated, efficient, and effective system of assessment.

[1]  Lisa K. Son,et al.  Learners’ choices and beliefs about self-testing , 2009, Memory.

[2]  Donald A. Schön.  The reflective practitioner : how professionals think in action , 1986 .

[3]  R. Hatala,et al.  A contemporary approach to validity arguments: a practical guide to Kane's framework , 2015, Medical education.

[4]  M. Scardamalia,et al.  Doctors’ perspectives on their innovations in daily practice: implications for knowledge building in health care , 2008, Medical education.

[5]  G. Cizek,et al.  Defining and distinguishing validity: interpretations of score meaning and justifications of test use. , 2012, Psychological methods.

[6]  Timothy J. Wood,et al.  A procedural skills OSCE: assessing technical and non-technical skills of internal medicine residents , 2015, Advances in health sciences education : theory and practice.

[7]  Tim Dornan,et al.  Outcome (competency) based education: an exploration of its origins, theoretical basis, and empirical evidence , 2012, Advances in Health Sciences Education.

[8]  G. Regehr,et al.  Putting the expert together again , 2011, Medical education.

[9]  R. Tamblyn,et al.  Predictors of inappropriate antibiotic prescribing among primary care physicians , 2007, Canadian Medical Association Journal.

[10]  M. Kane.  The Argument-Based Approach to Validation , 1990 .

[11]  D. Newble,et al.  The effect of assessments and examinations on the learning of medical students , 1983, Medical education.

[12]  G Bordage,et al.  The Medical Council of Canada's key features project: a more valid written examination of clinical decision‐making skills , 1995, Academic medicine : journal of the Association of American Medical Colleges.

[13]  K. A. Ericsson,et al.  Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. , 2004, Academic medicine : journal of the Association of American Medical Colleges.

[14]  J. Colliver.  Educational Theory and Medical Education Practice: A Cautionary Note for Medical School Faculty , 2002, Academic medicine : journal of the Association of American Medical Colleges.

[15]  K. Newell,et al.  Time scales in motor learning and development. , 2001, Psychological review.

[16]  Jeffrey D. Karpicke,et al.  The Critical Importance of Retrieval for Learning , 2008, Science.

[17]  Charles B. Fleming,et al.  Opening the Black Box: Using Process Evaluation Measures to Assess Implementation and Theory Building , 1999, American journal of community psychology.

[18]  M. Mylopoulos,et al.  When I say … adaptive expertise , 2017, Medical education.

[19]  Jonathan Sherbino,et al.  The role of assessment in competency-based medical education , 2010, Medical teacher.

[20]  Robyn Tamblyn,et al.  Doctor scores on national qualifying examinations predict quality of care in future practice , 2009, Medical education.

[21]  B. Hodges.  OSCE! Variations on a theme by Harden , 2003, Medical education.

[22]  Geoff Norman,et al.  The minimal relationship between simulation fidelity and transfer of learning , 2012, Medical education.

[23]  Mark J. Gierl,et al.  Evaluating the quality of medical multiple‐choice items created with automated processes , 2013, Medical education.

[24]  R. Hays,et al.  Reflection or ‘pre‐reflection’: what are we actually measuring in reflective practice? , 2011, Medical education.

[25]  Kevin W Eva,et al.  The Aging Physician: Changes in Cognitive Processing and Their Impact on Medical Practice , 2002, Academic medicine : journal of the Association of American Medical Colleges.

[26]  Adam Dubrowski,et al.  The application of the challenge point framework in medical education , 2012, Medical education.

[27]  T. Dornan,et al.  Features of assessment learners use to make informed self‐assessments of clinical performance , 2011, Medical education.

[28]  S. Downing.  Validity: on the meaningful interpretation of assessment data , 2003, Medical education.

[29]  Michael T. Kane,et al.  An argument-based approach to validity. , 1992 .

[30]  V. Shute.  Focus on Formative Feedback , 2008 .

[31]  J. Norcini,et al.  Workplace-based assessment as an educational tool: AMEE Guide No. 31 , 2007, Medical teacher.

[32]  M. Pusic,et al.  Developing the role of big data and analytics in health professional education , 2014, Medical teacher.

[33]  D. Schoen,et al.  The Reflective Practitioner: How Professionals Think in Action , 1985 .

[34]  J. Norcini.  Current perspectives in assessment: the assessment of performance at work , 2005, Medical education.

[35]  L. Lingard,et al.  Learning culture and feedback: an international study of medical athletes and musicians , 2014, Medical education.

[36]  Gordon Page,et al.  A practical guide to assessing clinical decision‐making skills using the key features approach , 2005, Medical education.

[37]  R. Hatala,et al.  Modification of an OSCE format to enhance patient continuity in a high-stakes assessment of clinical performance , 2011, BMC medical education.

[38]  G. Regehr,et al.  Toward Authentic Clinical Evaluation: Pitfalls in the Pursuit of Competency , 2010, Academic medicine : journal of the Association of American Medical Colleges.

[39]  Cees P. M. van der Vleuten,et al.  Assessing professional competence: from methods to programmes , 2005 .

[40]  Glenn Regehr,et al.  Coaching Surgeons: Is Culture Limiting Our Ability to Improve? , 2015, Annals of surgery.

[41]  H. Marsh,et al.  Making students' evaluations of teaching effectiveness effective: The critical issues of validity, bias, and utility. , 1997 .

[42]  Karen D. Könings,et al.  Barriers to the uptake and use of feedback in the context of summative assessment , 2015, Advances in health sciences education : theory and practice.

[43]  R. Galbraith,et al.  Conceptual Perspectives: Emerging Changes in the Assessment Paradigm , 2011 .

[44]  Vanessa C. Burch,et al.  Workplace-based assessment as an educational tool , 2007 .

[45]  Mark J. Gierl,et al.  Using automatic item generation to create multiple‐choice test items , 2012, Medical education.

[46]  L. Kohn,et al.  To Err Is Human : Building a Safer Health System , 2007 .

[47]  J. Kogan,et al.  Realizing the Promise and Importance of Performance-Based Assessment , 2013, Teaching and learning in medicine.

[48]  P. Yeates,et al.  Seeing the ‘black box’ differently: assessor cognition from three research perspectives , 2014, Medical education.

[49]  Geoff Norman,et al.  Competency-based education: milestones or millstones? , 2014, Journal of graduate medical education.

[50]  Robert Fletcher,et al.  Systematic Review: The Relationship between Clinical Experience and Quality of Health Care , 2005, Annals of Internal Medicine.

[51]  R. Duvivier,et al.  Criteria for good assessment: Consensus statement and recommendations from the Ottawa 2010 Conference , 2011, Medical teacher.

[52]  K. Mann,et al.  Reflection and reflective practice in health professions education: a systematic review , 2009, Advances in health sciences education : theory and practice.

[53]  R. Reznick,et al.  A new assessment tool: the patient assessment and management examination. , 1997, Surgery.

[54]  Reed G. Williams,et al.  Variations in Senior Medical Student Diagnostic Justification Ability , 2014, Academic medicine : journal of the Association of American Medical Colleges.

[55]  C. V. D. Vleuten,et al.  The assessment of professional competence: Developments, research and practical implications , 1996 .

[56]  R. Tamblyn,et al.  Practice Indicators of Suboptimal Care and Avoidable Adverse Events: A Content Analysis of a National Qualifying Examination , 2013, Academic medicine : journal of the Association of American Medical Colleges.

[57]  G. Regehr,et al.  ‘It’s a cultural expectation…’ The pressure on medical trainees to work independently in clinical practice , 2009, Medical education.

[58]  P. Teunissen,et al.  Opportunity or threat: the ambiguity of the consequences of transitions in medical education , 2011, Medical education.

[59]  K. Eva,et al.  Scylla or Charybdis? Can we navigate between objectification and judgement in assessment? , 2012, Medical education.

[60]  Eric S. Holmboe,et al.  Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. , 2009, JAMA.

[61]  R. Bjork.  Memory and metamemory considerations in the training of human beings. , 1994 .

[62]  A. Kluger,et al.  Feedback, the various tasks of the doctor, and the feedforward alternative , 2010, Medical education.

[63]  C. Ringsted,et al.  The testing effect on skills learning might last 6 months , 2010, Advances in health sciences education : theory and practice.

[64]  When I say … capability , 2016, Medical education.

[65]  D. Klass.  A performance-based conception of competence is changing the regulation of physicians' professional behavior. , 2007, Academic medicine : journal of the Association of American Medical Colleges.

[66]  G. Regehr,et al.  Effective feedback for maintenance of competence: from data delivery to trusting dialogues , 2013, Canadian Medical Association Journal.

[67]  Douglas P Larsen,et al.  Test‐enhanced learning in medical education , 2008, Medical education.

[68]  T. Roberts,et al.  Trends in national licensing examinations in medicine , 2016, Medical education.

[69]  Lorelei Lingard,et al.  Basing the Evaluation of Professionalism on Observable Behaviors: A Cautionary Tale , 2004, Academic medicine : journal of the Association of American Medical Colleges.

[70]  David L Buckeridge,et al.  Physician scores on a national clinical skills examination as predictors of complaints to medical regulatory authorities. , 2007, JAMA.

[71]  K. Eva,et al.  Which Factors, Personal or External, Most Influence Students' Generation of Learning Goals? , 2010, Academic medicine : journal of the Association of American Medical Colleges.

[72]  David Boud,et al.  Feedback in Higher and Professional Education : Understanding it and doing it well , 2013 .

[73]  J. Gold,et al.  The ABMS MOC Part III Examination: Value, Concerns, and Alternative Formats , 2016, Academic medicine : journal of the Association of American Medical Colleges.

[74]  K. Eva.  On the generality of specificity , 2003, Medical education.

[75]  Glenn Regehr,et al.  Competency is not enough: integrating identity formation into the medical education discourse. , 2012, Academic medicine : journal of the Association of American Medical Colleges.

[76]  G. Regehr,et al.  Do In-Training Evaluation Reports Deserve Their Bad Reputations? A Study of the Reliability and Predictive Ability of ITER Scores and Narrative Comments , 2013, Academic medicine : journal of the Association of American Medical Colleges.

[77]  R. Cruess,et al.  Updating the Hippocratic Oath to include medicine's social contract , 2014, Medical education.

[78]  Kevin W Eva,et al.  Diagnostic error in medical education: where wrongs can make rights , 2009, Advances in health sciences education : theory and practice.

[79]  G. Regehr,et al.  ADAPTING OBJECTIVE STRUCTURED CLINICAL EXAMINATIONS TO ASSESS SOCIAL WORK STUDENTS' PERFORMANCE AND REFLECTIONS , 2011 .

[80]  K. Eva,et al.  The difficulty with experience: Does practice increase susceptibility to premature closure? , 2006, The Journal of continuing education in the health professions.

[81]  Joan Sargeant,et al.  Tensions in Informed Self-Assessment: How the Desire for Feedback and Reticence to Collect and Use It Can Conflict , 2011, Academic medicine : journal of the Association of American Medical Colleges.

[82]  E. Custers.  Long-term retention of basic science knowledge: a review study , 2010, Advances in health sciences education : theory and practice.

[83]  Erik W Driessen,et al.  Portfolios for assessment and learning: AMEE Guide no. 45 , 2009, Medical teacher.

[84]  D. Long,et al.  Competency-based medical education: theory to practice , 2010, Medical teacher.

[85]  G. Bordage,et al.  Content and Conceptual Frameworks of Preceptor Feedback Related to Residents’ Educational Needs , 2012, Academic medicine : journal of the Association of American Medical Colleges.

[86]  H. Pashler,et al.  Recent Research on Human Learning Challenges Conventional Instructional Strategies , 2010 .

[87]  J. P. Minda,et al.  Developing a Unified List of Physicians’ Reasoning Tasks During Clinical Encounters , 2013, Academic medicine : journal of the Association of American Medical Colleges.

[88]  J. Norcini,et al.  The Mini-CEX: A Method for Assessing Clinical Skills , 2003, Annals of Internal Medicine.

[89]  J. Kogan,et al.  Opening the black box of clinical skills assessment via observation: a conceptual model , 2011, Medical education.

[90]  E. Holmboe,et al.  Making self-assessment more effective. , 2008, The Journal of continuing education in the health professions.

[91]  J. James.  A New, Evidence-based Estimate of Patient Harms Associated with Hospital Care , 2013, Journal of patient safety.

[92]  W. Iobst,et al.  Optimizing the implementation of practice improvement modules in training: lessons from educators. , 2013, Journal of graduate medical education.

[93]  R. Butler.  Task-involving and ego-involving properties of evaluation: Effects of different feedback conditions on motivational perceptions, interest, and performance. , 1987 .

[94]  A. Neville,et al.  Assessment steers learning down the right road: Impact of progress testing on licensing examination performance , 2010, Medical teacher.

[95]  Brian Hodges,et al.  Seeking inclusion in an exclusive process: discourses of medical school student selection , 2015, Medical education.

[96]  D. Cook.  When I say… validity , 2014, Medical education.