Mapping cognitive overlaps between practice-based learning and improvement and evidence-based medicine: an operational definition for assessing resident physician competence.

PURPOSE The complex competency labeled practice-based learning and improvement (PBLI) by the Accreditation Council for Graduate Medical Education (ACGME) incorporates core knowledge in evidence-based medicine (EBM). The purpose of this study was to operationally define a "PBLI-EBM" domain for assessing resident physician competence. METHOD The authors used an iterative design process to first content analyze and map correspondences between ACGME and EBM literature sources. The project team, including content and measurement experts and residents/fellows, parsed, classified, and hierarchically organized embedded learning outcomes using a literature-supported cognitive taxonomy. A pool of 141 items was produced from the domain and assessment specifications. The PBLI-EBM domain and resulting items were content validated through formal reviews by a national panel of experts. RESULTS The final domain represents overlapping PBLI and EBM cognitive dimensions measurable through written, multiple-choice assessments. It is organized as 4 subdomains of clinical action: Therapy, Prognosis, Diagnosis, and Harm. Four broad cognitive skill branches (Ask, Acquire, Appraise, and Apply) are subsumed under each subdomain. Each skill branch is defined by enabling skills that specify the cognitive processes, content, and conditions pertinent to demonstrable competence. Most items passed content validity screening criteria and were prepared for test form assembly and administration. CONCLUSIONS The operational definition of PBLI-EBM competence is based on a rigorously developed and validated domain and item pool, and substantially expands conventional understandings of EBM. The domain, assessment specifications, and procedures outlined may be used to design written assessments to tap important cognitive dimensions of the overall PBLI competency, as given by ACGME. For more comprehensive coverage of the PBLI competency, such instruments need to be complemented with performance assessments.

[1]  A. Haig,et al.  The effectiveness of portfolios for post-graduate assessment and education: BEME Guide No 12 , 2009, Medical teacher.

[2]  Gordon H. Guyatt,et al.  Users' guides to the medical literature. II. How to use an article about therapy or prevention. A. Are the results of the study valid? Evidence-Based Medicine Working Group. , 1993, JAMA.

[3]  S. Lurie,et al.  Measurement of the General Competencies of the Accreditation Council for Graduate Medical Education: A Systematic Review , 2009, Academic medicine : journal of the Association of American Medical Colleges.

[4]  John M. Ferron,et al.  Using an Iterative Model to Conceptualize, Pilot Test, and Validate Scores from an Instrument Measuring Teacher Readiness for Educational Reforms , 2002 .

[5]  D. Bell,et al.  Instruments for evaluating education in evidence-based practice: a systematic review. , 2006, JAMA.

[6]  D. Sackett.  Evidence-Based Medicine: How to Practice and Teach EBM , 2018 .

[7]  Luc Goossens,et al.  Concurrent and Predictive Validity of the Student Adaptation to College Questionnaire in a Sample of European Freshman Students , 2002 .

[8]  P. R. Manning.  Practice-based learning and improvement: a dream that can become a reality. , 2003, The Journal of continuing education in the health professions.

[9]  Teaching Evidence-Based Medicine to Internal Medicine Residents: The Efficacy of Conferences Versus Small-Group Discussion , 2005, Teaching and learning in medicine.

[10]  Eduardo Ortiz,et al.  Evaluating the teaching of evidence based medicine: conceptual framework , 2004, BMJ : British Medical Journal.

[11]  R. Glaser,et al.  Knowing What Students Know: The Science and Design of Educational Assessment , 2001 .

[12]  L. Crocker,et al.  Introduction to Classical and Modern Test Theory , 1986 .

[13]  Jay McTighe,et al.  Backward Design for Forward Action. , 2003 .

[14]  Sharon E Straus,et al.  Evidence-Based Medicine: How to Practice and Teach It , 2010 .

[15]  Prathibha Varkey,et al.  A systems approach for implementing practice-based learning and improvement and systems-based practice in graduate medical education. , 2009, Academic medicine : journal of the Association of American Medical Colleges.

[16]  T. Lesnick,et al.  Validity Evidence for an OSCE to Assess Competency in Systems-Based Practice and Practice-Based Learning and Improvement: A Preliminary Investigation , 2008, Academic medicine : journal of the Association of American Medical Colleges.

[17]  K. Ramos,et al.  Validation of the Fresno test of competence in evidence based medicine , 2003, BMJ : British Medical Journal.

[18]  B. Rowe,et al.  A survey to determine the prevalence and characteristics of training in Evidence-Based Medicine in emergency medicine residency programs , 2005 .

[19]  J. J. Celentano,et al.  Do workshops in evidence-based practice equip participants to identify and answer questions requiring consideration of clinical research? A diagnostic skill assessment , 2009, Advances in health sciences education : theory and practice.

[20]  S. Swing,et al.  The ACGME outcome project: retrospective and prospective , 2007, Medical teacher.

[21]  Gordon H. Guyatt,et al.  Users' Guides to the Medical Literature: A Manual for Evidence-Based Clinical Practice; Users' Guides to the Medical Literature: Essentials of Evidence-Based Clinical Practice , 2003, BMJ : British Medical Journal.

[22]  Peter C Wyer,et al.  Where is the wisdom? I--a conceptual history of evidence-based medicine. , 2009, Journal of evaluation in clinical practice.

[23]  Madhabi Chatterji,et al.  Systems-based practice defined: taxonomy development and role identification for competency assessment of residents. , 2009, Journal of graduate medical education.

[24]  T. Greenhalgh,et al.  Do short courses in evidence based medicine improve knowledge and skills? Validation of Berlin questionnaire and before and after study of courses in evidence based medicine , 2002, BMJ : British Medical Journal.

[25]  L. Headrick,et al.  Continuous improvement learning for residents. , 1998, Pediatrics.

[26]  Jay McTighe,et al.  Assessing Student Outcomes: Performance Assessment Using the Dimensions of Learning Model , 1993 .

[27]  H. Bernard,et al.  Data Management and Analysis Methods , 2000 .

[28]  Benjamin S. Bloom,et al.  Taxonomy of Educational Objectives: The Classification of Educational Goals. , 1957 .

[29]  J. Millman,et al.  The specification and development of tests of achievement and ability. , 1989 .