Understanding the Assessment of Clinical Reasoning

Clinical reasoning assessment is an essential component of determining a health professional’s competence. Clinical reasoning cannot be assessed directly. It must be gleaned from a health professional’s choices and decisions. Clinical knowledge and knowledge organization, rather than a general problem solving process, serve as the substrate for clinical reasoning ability. Unfortunately, the lack of a gold standard for the clinical reasoning process and the observation of context specificity make it difficult to assess clinical reasoning. Information processing theory, which focuses on the way the brain processes and organizes knowledge, has provided valuable insights into the cognitive psychology of diagnostic and therapeutic reasoning but failed to explain the variance in health professional’s diagnostic performance. Situativity theory has emerged suggesting that this variance relates to context-specific factors that impact a health professional’s clinical reasoning performance. Both information processing and situativity theory inform the way in which we assess clinical reasoning. Current assessment methods focus on standardized testing of knowledge to maximize psychometric parameters and work-based assessments which evaluate clinical reasoning under authentic, uncertain conditions that can decrease the reliability of these measurements. Issues of inter-rater reliability and context specificity require that multiple raters assess multiple encounters in multiple contexts to optimize validity and reliability. No single assessment method can assess all aspects of clinical reasoning; therefore, in order to improve the quality of assessments of clinical reasoning ability, different combinations of methods that measure different components of the clinical reasoning process are needed.

[1]  J. Sandars,et al.  Perspective: viewing "strugglers" through a different lens: how a self-regulated learning perspective can help medical educators with assessment and remediation. , 2011, Academic medicine : journal of the Association of American Medical Colleges.

[2]  J. Norcini,et al.  The Mini-CEX: A Method for Assessing Clinical Skills , 2003, Annals of Internal Medicine.

[3]  P. Croskerry The Importance of Cognitive Errors in Diagnosis and Strategies to Minimize Them , 2003, Academic medicine : journal of the Association of American Medical Colleges.

[4]  J. Norcini,et al.  Assessment in the Context of Licensure and Certification , 2013, Teaching and learning in medicine.

[5]  Cees P. M. van der Vleuten,et al.  Assessing professional competence: from methods to programmes , 2005 .

[6]  S. Durning,et al.  Clarifying assumptions to enhance our understanding and assessment of clinical reasoning. , 2013, Academic medicine : journal of the Association of American Medical Colleges.

[7]  C. Hui-Chan,et al.  Clinical reasoning and the occupational therapy curriculum , 2000 .

[8]  K. A. Ericsson,et al.  An expert‐performance perspective of research on medical expertise: the study of clinical performance , 2007, Medical education.

[9]  Steven J Durning,et al.  Assessing clinical reasoning: moving from in vitro to in vivo , 2014, Diagnosis.

[10]  Keiko Abe,et al.  The use of simulated patients in medical education: AMEE Guide No 42 , 2009, Medical teacher.

[11]  Megan Smith,et al.  Clinical reasoning in physiotherapy , 2008 .

[12]  J. Higgs,et al.  Clinical Reasoning in the Health Professions , 2000 .

[13]  S. Haist,et al.  The relationship between performance on the Infectious Diseases In-Training and Certification Examinations. , 2015, Clinical infectious diseases : an official publication of the Infectious Diseases Society of America.

[14]  R. Lipner,et al.  American Board of Medical Specialties Maintenance of Certification: Theory and Evidence Regarding the Current Framework , 2013, The Journal of continuing education in the health professions.

[15]  L D Gruppen,et al.  Information Gathering and Integration as Sources of Error in Diagnostic Decision Making , 1991, Medical decision making : an international journal of the Society for Medical Decision Making.

[16]  G. Norman,et al.  Conceptual and methodological issues in studies comparing assessment formats , 1996 .

[17]  B. Hodges Assessment in the post-psychometric era: Learning to love the subjective and collective , 2013, Medical teacher.

[18]  A. Tversky,et al.  Judgment under Uncertainty , 1982 .

[19]  G. Bordage Prototypes and semantic qualifiers: from past to present , 2007, Medical education.

[20]  Douglas P Larsen,et al.  Test‐enhanced learning in medical education , 2008, Medical education.

[21]  Michelle E. Costanzo,et al.  Using functional magnetic resonance imaging to improve how we understand, teach, and assess clinical reasoning. , 2014, The Journal of continuing education in the health professions.

[22]  C. Kreiter,et al.  The Psychometric Properties of Five Scoring Methods Applied to the Script Concordance Test , 2005, Academic medicine : journal of the Association of American Medical Colleges.

[23]  L. Shulman,et al.  Medical Problem Solving: An Analysis of Clinical Reasoning , 1978 .

[24]  Jeffrey D. Karpicke,et al.  Retrieval Practice Produces More Learning than Elaborative Studying with Concept Mapping , 2011, Science.

[25]  G Bordage,et al.  The Medical Council of Canada's key features project: a more valid written examination of clinical decision‐making skills , 1995, Academic medicine : journal of the Association of American Medical Colleges.

[26]  G. Sugiyama,et al.  Study habits centered on completing review questions result in quantitatively higher American Board of Surgery In-Training Exam scores. , 2014, Journal of surgical education.

[27]  Eric S. Holmboe,et al.  Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. , 2009, JAMA.

[28]  B. Charlin,et al.  Script concordance testing: From theory to practice: AMEE Guide No. 75 , 2013, Medical teacher.

[29]  C. V. D. Vleuten,et al.  The assessment of professional competence: Developments, research and practical implications , 1996 .

[30]  J. Gonnella,et al.  The Simulated Patient Management Problem: a Technique for Evaluating and Teaching Clinical Competence , 1967, British journal of medical education.

[31]  F. Paas,et al.  Cognitive Architecture and Instructional Design , 1998 .

[32]  J. Sweller,et al.  Cognitive Load Theory and Complex Learning: Recent Developments and Future Directions , 2005 .

[33]  Eleanor Rosch,et al.  Principles of Categorization , 1978 .

[34]  A. Avins,et al.  Do ratings on the american board of internal medicine resident evaluation form detect differences in clinical competence? , 1994, Journal of General Internal Medicine.

[35]  E. Holmboe,et al.  Methods for Evaluating the Clinical Competence of Residents in Internal Medicine: A Review , 1998, Annals of Internal Medicine.

[36]  E. Holmboe,et al.  Teaching and evaluating point of care learning with an Internet‐based clinical‐question portfolio , 2009, The Journal of continuing education in the health professions.

[37]  L. Schuwirth,et al.  A plea for new psychometric models in educational assessment , 2006, Medical education.

[38]  Allen Newell,et al.  Human Problem Solving. , 1973 .

[39]  Candace Schau,et al.  Use of Map Techniques in Teaching Applied Statistics Courses , 1997 .

[40]  Henk G. Schmidt,et al.  Conscious thought beats deliberation without attention in diagnostic decision-making: at least when you are an expert , 2010, Psychological research.

[41]  U Fors,et al.  The use of a virtual patient case in an OSCE-based exam – A pilot study , 2008, Medical teacher.

[42]  E. Holmboe,et al.  The in-training examination in internal medicine: resident perceptions and lack of correlation between resident scores and faculty predictions of resident performance. , 1999, The American journal of medicine.

[43]  J. Hattie,et al.  The Power of Feedback , 2007 .

[44]  Gary Klein,et al.  Sources of Power: How People Make Decisions , 2017 .

[45]  S. Ross,et al.  Structured teaching and assessment: a new chart-stimulated recall worksheet for family medicine residents. , 2010, Canadian family physician Medecin de famille canadien.

[46]  R. Evans,et al.  Reliability of the "Draw-a-Man" test. , 1975 .

[47]  Henk G Schmidt,et al.  How expertise develops in medicine: knowledge encapsulation and illness script formation , 2007, Medical education.

[48]  A. Greiner,et al.  Assessing quality of care: knowledge matters. , 2008, JAMA.

[49]  G. A. Miller THE PSYCHOLOGICAL REVIEW THE MAGICAL NUMBER SEVEN, PLUS OR MINUS TWO: SOME LIMITS ON OUR CAPACITY FOR PROCESSING INFORMATION 1 , 1956 .

[50]  C. Walsh,et al.  Virtual reality simulation training for health professions trainees in gastrointestinal endoscopy. , 2012, The Cochrane database of systematic reviews.

[51]  G. Salomon Distributed cognitions : psychological and educational considerations , 1997 .

[52]  K. Lasater Clinical judgment development: using simulation to create an assessment rubric. , 2007, The Journal of nursing education.

[53]  Vimla L. Patel,et al.  Avatar-based simulation in the evaluation of diagnosis and management of mental health disorders in primary care , 2012, J. Biomed. Informatics.

[54]  G R Norman,et al.  Exploring the etiology of content specificity: factors influencing analogic transfer and problem solving. , 1998, Academic medicine : journal of the Association of American Medical Colleges.

[55]  E. Hanna,et al.  Defensible assessment of the competency of the practicing physician , 1997, Academic medicine : journal of the Association of American Medical Colleges.

[56]  K. Eva What every teacher needs to know about clinical reasoning , 2005, Medical education.

[57]  C. Violato,et al.  Assessment of physician performance in Alberta: the physician achievement review. , 1999, CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne.

[58]  L. Roberts,et al.  Using concept maps to measure statistical understanding , 1999 .

[59]  Gordon Page,et al.  A practical guide to assessing clinical decision‐making skills using the key features approach , 2005, Medical education.

[60]  M. Fonteyn,et al.  Expert nurses' clinical reasoning under uncertainty: representation, structure, and process. , 1992, Proceedings. Symposium on Computer Applications in Medical Care.

[61]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[62]  R. Lipner,et al.  Assessing Quality of Care , 2008 .

[63]  Paul J. Feltovich,et al.  The Cambridge handbook of expertise and expert performance , 2006 .

[64]  Ryan Brydges,et al.  A reflective analysis of medical education research on self‐regulation in learning and practice , 2012, Medical education.

[65]  Barry J. Zimmerman,et al.  Self-Regulated Learning , 2001 .

[66]  G. Bordage,et al.  Script concordance tests: strong inferences about examinees require stronger evidence , 2014, Medical education.

[67]  G. Webster,et al.  Reliability, validity and efficiency of various item formats in assessment of physician competence. , 1983, Proceedings of the ... annual Conference on Research in Medical Education. Conference on Research in Medical Education.

[68]  G. Gorry,et al.  Towards the simulation of clinical cognition. Taking a present illness by computer. , 1976, The American journal of medicine.

[69]  K. Lasater Clinical judgment: the last frontier for evaluation. , 2011, Nurse education in practice.

[70]  S. Durning,et al.  Situativity theory: A perspective on how participants and the environment can interact: AMEE Guide no. 52 , 2011, Medical teacher.

[71]  L. Schuwirth Is assessment of clinical reasoning still the Holy Grail? , 2009, Medical education.

[72]  D. Kahneman Thinking, Fast and Slow , 2011 .

[73]  K. Adamson,et al.  Assessing the reliability, validity, and use of the Lasater Clinical Judgment Rubric: three approaches. , 2012, The Journal of nursing education.

[74]  N. Franklin,et al.  Diagnostic error in internal medicine. , 2005, Archives of internal medicine.

[75]  F. Goulet,et al.  Influence of remedial professional development programs for poorly performing physicians , 2007, The Journal of continuing education in the health professions.

[76]  G. Miller The assessment of clinical skills/competence/performance , 1990, Academic medicine : journal of the Association of American Medical Colleges.

[77]  T. Walsh,et al.  Exercise rehabilitation following intensive care unit discharge for recovery from critical illness. , 2015, The Cochrane database of systematic reviews.

[78]  David B. Swanson,et al.  Extended‐matching items: A practical alternative to free‐response questions , 1993 .

[79]  Eric S. Holmboe,et al.  Faculty and the Observation of Trainees’ Clinical Skills: Problems and Opportunities , 2004, Academic medicine : journal of the Association of American Medical Colleges.

[80]  G. Webster,et al.  A Comparison of Knowledge, Synthesis, and Clinical Judgment , 1984, Evaluation & the health professions.

[81]  Kevin W Eva,et al.  The benefits of flexibility: the pedagogical value of instructions to adopt multifaceted diagnostic reasoning strategies , 2007, Medical education.

[82]  S. Downing,et al.  Reliability and Validity of Key Feature Cases for the Self-Assessment of Colon and Rectal Surgeons , 2008, Annals of surgery.

[83]  Kevin W Eva,et al.  Rater-Based Assessments as Social Judgments: Rethinking the Etiology of Rater Errors , 2011, Academic medicine : journal of the Association of American Medical Colleges.

[84]  D. Boud,et al.  Rethinking models of feedback for learning: the challenge of design , 2013 .

[85]  G. Bordage,et al.  Threats to validity in the use and interpretation of script concordance test scores , 2013, Medical education.

[86]  E. Holmboe,et al.  Comprar Practical Guide to the Evaluation of Clinical Competence with bonus DVD | Eric S. Holmboe | 9780323047098 | Mosby , 2008 .

[87]  A. Muijtjens,et al.  Workplace-based assessment: effects of rater expertise , 2010, Advances in health sciences education : theory and practice.

[88]  John B. Black,et al.  The representation of scripts in memory , 1985 .

[89]  H. Rimoldi Rationale and applications of the test of diagnostic skills. , 1963, Journal of medical education.

[90]  R. Duvivier,et al.  Criteria for good assessment: Consensus statement and recommendations from the Ottawa 2010 Conference , 2011, Medical teacher.

[91]  D. Krathwohl A Revision of Bloom's Taxonomy: An Overview , 2002 .

[92]  A. Collins,et al.  Situated Cognition and the Culture of Learning , 1989 .

[93]  J. Lave Cognition in practice , 1988 .

[94]  Bernard Charlin,et al.  Scripts and clinical reasoning , 2007, Medical education.

[95]  B. Zimmerman Motivational Sources and Outcomes of Self-Regulated Learning and Performance , 2011 .

[96]  B. Daley,et al.  Concept maps in medical education: an analytical literature review , 2010, Medical education.

[97]  B. Charlin,et al.  How to construct and implement script concordance tests: insights from a systematic review , 2012, Medical education.

[98]  Sorabh Khandelwal,et al.  The 2013 Model of the Clinical Practice of Emergency Medicine. , 2014, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[99]  G. Podgorny,et al.  The certification examination in emergency medicine: an update. , 1982, Annals of emergency medicine.

[100]  S. Babbott,et al.  The predictive validity of the internal medicine in-training examination. , 2007, The American journal of medicine.

[101]  B. Daley,et al.  A qualitative evaluation of medical student learning with concept maps , 2007, Medical teacher.

[102]  K. Eva On the generality of specificity , 2003, Medical education.

[103]  Vicki R Leblanc,et al.  Teaching from the clinical reasoning literature: combined reasoning strategies help novice diagnosticians overcome misleading information , 2007, Medical education.

[104]  J. Kogan,et al.  What Drives Faculty Ratings of Residents' Clinical Skills? The Impact of Faculty's Own Clinical Skills , 2010, Academic medicine : journal of the Association of American Medical Colleges.

[105]  A. Jefferies,et al.  Assessment of multiple physician competencies in postgraduate training: utility of the structured oral examination , 2011, Advances in health sciences education : theory and practice.

[106]  M. Araceli Ruiz-Primo,et al.  Examining Concept Maps as an Assessment Tool , 2004 .

[107]  Bernard Charlin,et al.  Standardized Assessment of Reasoning in Contexts of Uncertainty , 2004, Evaluation & the health professions.