Developing a dashboard to meet Competence Committee needs: a design-based research project

Background Competency-based programs are being adopted in medical education around the world. Competence Committees must visualize learner assessment data effectively to support their decision-making. Dashboards play an integral role in decision support systems in other fields. Design-based research allows the simultaneous development and study of educational environments. Methods We utilized a design-based research process within the emergency medicine residency program at the University of Saskatchewan to identify the data, analytics, and visualizations needed by its Competence Committee, and developed a dashboard incorporating these elements. Narrative data were collected from two focus groups, five interviews, and the observation of two Competence Committee meetings. Data were qualitatively analyzed to develop a thematic framework outlining the needs of the Competence Committee and to inform the development of the dashboard. Results The qualitative analysis identified four Competence Committee needs (Explore Workplace-Based Assessment Data, Explore Other Assessment Data, Understand the Data in Context, and Ensure the Security of the Data). These needs were described with narratives and represented through visualizations of the dashboard elements. Conclusions This work addresses the practical challenges of supporting data-driven decision making by Competence Committees and will inform the development of dashboards for programs, institutions, and learner management systems.

[1]  M. Pusic,et al.  Developing the role of big data and analytics in health professional education , 2014, Medical teacher.

[2]  Diana H J M Dolmans,et al.  Building bridges between theory and practice in medical education using a design-based research approach: AMEE Guide No. 60 , 2012, Medical teacher.

[3]  Linda Corrin,et al.  Learning Analytics: A Case Study of the Process of Design of Visualizations , 2012 .

[4]  Jonathan Sherbino,et al.  The McMaster Modular Assessment Program (McMAP): A Theoretically Grounded Work-Based Assessment System for an Emergency Medicine Residency Program , 2015, Academic medicine : journal of the Association of American Medical Colleges.

[5]  Mostafa Langarizadeh,et al.  Evaluation of Effective Dashboards: Key Concepts and Criteria , 2017, The open medical informatics journal.

[6]  S. Hamstra,et al.  Entrustability Scales: Outlining Their Usefulness for Competency-Based Clinical Assessment , 2016, Academic medicine : journal of the Association of American Medical Colleges.

[7]  T. Chan,et al.  Attending Emergency Physicians’ Perceptions of a Programmatic Workplace-Based Assessment System: The McMaster Modular Assessment Program (McMAP) , 2019, Teaching and learning in medicine.

[8]  T. Chan,et al.  McMaster Modular Assessment Program (McMAP) Through the Years: Residents' Experience With an Evolving Feedback Culture Over a 3‐year Period , 2017, AEM education and training.

[9]  B. Holroyd,et al.  The competency-based medical education evolution of Canadian emergency medicine specialist training. , 2019, CJEM.

[10]  Timothy J. Wood,et al.  Continued Validation of the O-SCORE (Ottawa Surgical Competency Operating Room Evaluation): Use in the Simulated Environment , 2016, Teaching and learning in medicine.

[11]  Jacqueline E McLaughlin,et al.  Design-based research: Connecting theory and practice in pharmacy educational intervention research. , 2019, Currents in pharmacy teaching & learning.

[12]  E. Warm,et al.  Twelve tips to maximize the value of a clinical competency committee in postgraduate medical education , 2018, Medical teacher.

[13]  Thomas C. Reeves,et al.  Design research: A socially responsible approach to instructional technology research in higher education , 2005, J. Comput. High. Educ..

[14]  Timothy J. Wood,et al.  The Ottawa Surgical Competency Operating Room Evaluation (O-SCORE): A Tool to Assess Surgical Competence , 2012, Academic medicine : journal of the Association of American Medical Colleges.

[15]  Michael Gottlieb,et al.  Applying Design Thinking Principles to Curricular Development in Medical Education , 2017, AEM education and training.

[16]  P. O'Sullivan,et al.  Ensuring Resident Competence: A Narrative Review of the Literature on Group Decision Making to Inform the Work of Clinical Competency Committees. , 2016, Journal of graduate medical education.

[17]  L. Lingard,et al.  Grounded theory in medical education research: AMEE Guide No. 70 , 2012, Medical teacher.

[18]  D. Stockley,et al.  A Core Components Framework for Evaluating Implementation of Competency-Based Medical Education Programs. , 2019, Academic medicine : journal of the Association of American Medical Colleges.

[19]  P. O'Sullivan,et al.  Reviewing Residents’ Competence: A Qualitative Study of the Role of Clinical Competency Committees in Performance Assessment , 2015, Academic medicine : journal of the Association of American Medical Colleges.

[20]  Stephanie Boehm,et al.  Information Dashboard Design The Effective Visual Communication Of Data , 2016 .

[21]  Beth King,et al.  Influence of Clinical Competency Committee Review Process on Summative Resident Assessment Decisions. , 2018, Journal of graduate medical education.

[22]  Sue E. Poynter,et al.  Thresholds and interpretations: How clinical competency committees identify pediatric residents with performance concerns , 2018, Medical teacher.

[23]  Saima I. Chaudhry,et al.  Resident dashboards: helping your clinical competency committee visualize trainees’ key performance indicators , 2016, Medical education online.

[24]  S. Reddy,et al.  How do clinical competency committees use different sources of data to assess residents’ performance on the internal medicine milestones?A mixed methods pilot study , 2017, Medical teacher.

[25]  Karen E Hauer,et al.  Twelve tips to promote successful development of a learner performance dashboard within a medical education program , 2017, Medical teacher.

[26]  Weichao Chen,et al.  Twelve tips for conducting educational design research in medical education , 2020, Medical teacher.

[27]  Fedde Scheele,et al.  Experiences with EPAs, potential benefits and pitfalls , 2014, Medical teacher.

[28]  Theodore J. Kopcha,et al.  Conducting educational design research , 2013 .

[29]  D. Long,et al.  Competency-based medical education: theory to practice , 2010, Medical teacher.

[30]  The Clinical Competency Committee in Adult Cardiothoracic Anesthesiology-Perspectives From Program Directors Around the United States. , 2019, Journal of cardiothoracic and vascular anesthesia.

[31]  From aggregation to interpretation: how assessors judge complex data in a competency-based portfolio , 2017, Advances in health sciences education : theory and practice.

[32]  Adam Cooper,et al.  Produced by Cetis for Jisc Analytics Series Definition and Essential Characteristics What Is Analytics? Definition and Essential Characteristics , 2022 .

[33]  P. Sainsbury,et al.  Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. , 2007, International journal for quality in health care : journal of the International Society for Quality in Health Care.

[34]  A. Wise,et al.  Learning Analytics in Medical Education Assessment: The Past, the Present, and the Future , 2018, AEM education and training.

[35]  John P. Campbell,et al.  Academic Analytics: A New Tool for a New Era. , 2007 .

[36]  Jonathan Sherbino,et al.  The role of assessment in competency-based medical education , 2010, Medical teacher.

[37]  T. Chan,et al.  Nuance and Noise: Lessons Learned From Longitudinal Aggregated Assessment Data. , 2017, Journal of graduate medical education.

[38]  J. Frank,et al.  A call to action: The controversy of and rationale for competency-based medical education , 2017, Medical teacher.

[39]  L. Lingard,et al.  Some assembly required: tracing the interpretative work of Clinical Competency Committees , 2019, Medical education.