Usability evaluation of a medication reconciliation tool: Embedding safety probes to assess users' detection of medication discrepancies

OBJECTIVE An electronic medication reconciliation tool was previously developed by another research team to aid provider-patient communication for medication reconciliation. To evaluate the usability of this tool, we integrated artificial safety probes into standard usability methods. The objective of this article is to describe this method of using safety probes, which enabled us to evaluate how well the tool supports users' detection of medication discrepancies. MATERIALS AND METHODS We completed a mixed-method usability evaluation in a simulated setting with 30 participants: 20 healthcare professionals (HCPs) and 10 patients. We used factual scenarios but embedded three artificial safety probes: (1) a missing medication (i.e., omission); (2) an extraneous medication (i.e., commission); and (3) an inaccurate dose (i.e., dose discrepancy). We measured users' detection of each probe to estimate the probability that a HCP or patient would detect these discrepancies. Additionally, we recorded participants' detection of naturally occurring discrepancies. RESULTS Each safety probe was detected by ≤50% of HCPs. Patients' detection rates were generally higher. Estimates indicate that a HCP and patient, together, would detect 44.8% of these medication discrepancies. Additionally, HCPs and patients detected 25 and 45 naturally-occurring discrepancies, respectively. DISCUSSION Overall, detection of medication discrepancies was low. Findings indicate that more advanced interface designs are warranted. Future research is needed on how technologies can be designed to better aid HCPs' and patients' detection of medication discrepancies. CONCLUSION This is one of the first studies to evaluate the usability of a collaborative medication reconciliation tool and assess HCPs' and patients' detection of medication discrepancies. Results demonstrate that embedded safety probes can enhance standard usability methods by measuring additional, clinically-focused usability outcomes. The novel safety probes we used may serve as an initial, standard set for future medication reconciliation research. More prevalent use of safety probes could strengthen usability research for a variety of health information technologies.

[1]  Jeffrey L. Greenwald,et al.  Making inpatient medication reconciliation patient centered, clinically relevant and implementable: a consensus statement on key principles and necessary first steps. , 2010, Journal of hospital medicine.

[2]  Anestis Touloumis,et al.  GEE for Multinomial Responses Using a Local Odds Ratios Parameterization , 2013, Biometrics.

[3]  Vimla L. Patel,et al.  Cognitive and usability engineering methods for the evaluation of clinical information systems , 2004, J. Biomed. Informatics.

[4]  Andrew J. McLachlan,et al.  Impact of electronic medication reconciliation interventions on medication discrepancies at hospital transitions: a systematic review and meta-analysis , 2016, BMC Medical Informatics and Decision Making.

[5]  Ben Shneiderman,et al.  Novel user interface design for medication reconciliation: an evaluation of Twinlist , 2015, J. Am. Medical Informatics Assoc..

[6]  Jakob Nielsen,et al.  Estimating the number of subjects needed for a thinking aloud test , 1994, Int. J. Hum. Comput. Stud..

[7]  D. Bates,et al.  Adverse drug events occurring following hospital discharge , 2005, Journal of General Internal Medicine.

[8]  Monique W. M. Jaspers,et al.  A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence , 2009, Int. J. Medical Informatics.

[9]  Christopher M Callahan,et al.  Six-Item Screener to Identify Cognitive Impairment Among Potential Subjects for Clinical Research , 2002, Medical care.

[10]  J. Schnipper,et al.  The problem with medication reconciliation , 2016, BMJ Quality & Safety.

[11]  Alissa L. Russ,et al.  Design of a Medication Reconciliation Application , 2013, Applied Clinical Informatics.

[12]  Nicolette de Keizer,et al.  STARE-HI -Statement on Reporting of Evaluation Studies in Health Informatics , 2009, Yearbook of Medical Informatics.

[13]  Jan Horsky,et al.  Cognitive Errors in Reconciling Complex Medication Lists , 2016, AMIA.

[14]  Blake J. Lesselroth,et al.  Naturalistic Usability Testing of Inpatient Medication Reconciliation Software , 2017, ITCH.

[15]  J Brender,et al.  STARE-HI – Statement on Reporting of Evaluation Studies in Health Informatics , 2013, Applied Clinical Informatics.

[16]  Alissa L. Russ,et al.  Applying human factors principles to alert design increases efficiency and reduces prescribing errors in a scenario-based simulation. , 2014, Journal of the American Medical Informatics Association : JAMIA.

[17]  Leora I. Horwitz,et al.  Diagnoses and timing of 30-day readmissions after hospitalization for heart failure, acute myocardial infarction, or pneumonia. , 2013, JAMA.

[18]  Kathleen B Orrico.  Sources and Types of Discrepancies Between Electronic Medical Records and Actual Outpatient Medication Use , 2008, Journal of managed care pharmacy : JMCP.

[19]  Jeremy C. Wyatt,et al.  Making electronic prescribing alerts more effective: scenario-based experimental study in junior doctors , 2011, J. Am. Medical Informatics Assoc..

[20]  Becky L. Armor,et al.  Evaluation of Adverse Drug Events and Medication Discrepancies in Transitions of Care Between Hospital Discharge and Primary Care Follow-Up , 2016, Journal of pharmacy practice.

[21]  Blake J. Lesselroth,et al.  Using consumer-based kiosk technology to improve and standardize medication reconciliation in a specialty care setting. , 2009, Joint Commission journal on quality and patient safety.

[22]  André Kushniruk,et al.  Cognitive Analysis of a Medication Reconciliation Tool: Applying Laboratory and Naturalistic Approaches to System Evaluation , 2011, ITCH.

[23]  Rollin J Fairbanks,et al.  Electronic Health Record Vendor Adherence to Usability Certification Requirements and Testing Standards. , 2015, JAMA.

[24]  Anne Spinewine,et al.  Electronic tools to support medication reconciliation: a systematic review , 2017, J. Am. Medical Informatics Assoc..

[25]  Vimla L. Patel,et al.  Using usability heuristics to evaluate patient safety of medical devices , 2003, J. Biomed. Informatics.

[26]  Alissa L. Russ,et al.  Design and implementation of a hospital-based usability laboratory: insights from a Department of Veterans Affairs laboratory for health information technology. , 2012, Joint Commission journal on quality and patient safety.

[27]  James R. Lewis,et al.  IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use , 1995, Int. J. Hum. Comput. Interact..

[28]  T. Chen,et al.  The medication reconciliation process and classification of discrepancies: a systematic review. , 2016, British journal of clinical pharmacology.

[29]  Alissa L. Russ,et al.  Current Challenges and Opportunities for Better Integration of Human Factors Research with Development of Clinical Information Systems , 2009, Yearbook of Medical Informatics.

[30]  J. Cantero-Hinojosa,et al.  Medication reconciliation at admission and discharge: an analysis of prevalence and associated risk factors , 2015, International journal of clinical practice.

[31]  Catherine L. Liang,et al.  Effect of an electronic medication reconciliation application and process redesign on potential adverse drug events: a cluster-randomized trial. , 2009, Archives of internal medicine.

[32]  M. D. de Bruin‐Weller,et al.  Barriers and Facilitators to eHealth Use in Daily Practice: Perspectives of Patients and Professionals in Dermatology , 2017, Journal of medical Internet research.

[33]  C. Yancy,et al.  Relationship between early physician follow-up and 30-day readmission among Medicare beneficiaries hospitalized for heart failure. , 2010, JAMA.

[34]  Blake J. Lesselroth,et al.  Primary care provider perceptions and use of a novel medication reconciliation technology. , 2011, Informatics in primary care.

[35]  Jennifer A. Dunbar-Viveiros,et al.  In Situ Medical Simulation Investigation of Emergency Department Procedural Sedation With Randomized Trial of Experimental Bedside Clinical Process Guidance Intervention , 2015, Simulation in healthcare : journal of the Society for Simulation in Healthcare.

[36]  S. Simon,et al.  Engaging patients in medication reconciliation via a patient portal following hospital discharge. , 2014, Journal of the American Medical Informatics Association : JAMIA.

[37]  T. Sequist,et al.  Pharmacy Dispensing of Electronically Discontinued Medications , 2012, Annals of Internal Medicine.

[38]  Michelle L. Rogers,et al.  A simulation-based embedded probe technique for human-computer interaction evaluation , 2004, Cognition, Technology & Work.

[39]  Alissa L. Russ,et al.  Lessons learned from usability testing of the VA's personal health record , 2011, J. Am. Medical Informatics Assoc..

[40]  J. Nebeker,et al.  Medication reconciliation: barriers and facilitators from the perspectives of resident physicians and pharmacists. , 2011, Journal of hospital medicine.

[41]  Rebecca J Bartlett Ellis,et al.  Complexities of Medication Management Across Care Transitions: A Case Report , 2015, Clinical nurse specialist CNS.

[42]  P. Heidenreich,et al.  Dual health care system use is associated with higher rates of hospitalization and hospital readmission among veterans with heart failure. , 2016, American heart journal.

[43]  Alissa L. Russ,et al.  Ten factors to consider when developing usability scenarios and tasks for health information technology , 2018, J. Biomed. Informatics.