Determining differences in user performance between expert and novice primary care doctors when using an electronic health record (EHR).

RATIONALE, AIMS AND OBJECTIVES The goal of this study is to determine usability gaps between expert and novice primary care doctors when using an electronic health record (EHR). METHODS Usability tests using video analyses with triangular method approach were conducted to analyse usability gaps between 10 novice and seven expert doctors. Doctors completed 19 tasks, using think-aloud strategy, based on an artificial but typical patient visit note. The usability session lasted approximately 20 minutes. The testing room consisted of the participant and the facilitator. Mixed methods approach including four sets of performance measures, system usability scale (SUS), and debriefing session with participants was used. RESULTS While most expert doctors completed tasks more efficiently, and provided a higher SUS score than novice doctors (novice 68, expert 70 out of 100 being perfect score), the result of 'percent task success rate' were comparable (74% for expert group, 78% for novice group, P = 0.98) on all 19 tasks. CONCLUSION This study found a lack of expertise among doctors with more experience using the system demonstrating that although expert doctors have been using the system longer, their proficiency did not increase with EHR experience. These results may potentially improve the EHR training programme, which may increase doctors' performance when using an EHR. These results may also assist EHR vendors in improving the user interface, which may aid in reducing errors caused from poor usability of the system.

[1]  Emilie M. Roth,et al.  Research Paper: Impact of Clinical Reminder Redesign on Learnability, Efficiency, Usability, and Workload for Ambulatory Clinic Nurses , 2007, J. Am. Medical Informatics Assoc..

[2]  Suzanne Bakken,et al.  Review of health information technology usability study methodologies , 2011, J. Am. Medical Informatics Assoc..

[3]  Jan Stage,et al.  A longitudinal study of usability in health care: Does time heal? , 2010, Int. J. Medical Informatics.

[4]  Jeffery L. Belden,et al.  A Diabetes Dashboard and Physician Efficiency and Accuracy in Accessing Data Needed for High-Quality Diabetes Care , 2011, The Annals of Family Medicine.

[5]  J. P. Kichak,et al.  Computerized physician order entry: helpful or harmful? , 2003, Journal of the American Medical Informatics Association : JAMIA.

[6]  Kaveh G Shojania,et al.  Usability evaluation of order sets in a computerised provider order entry system , 2011, BMJ quality & safety.

[7]  Anne Holbrook,et al.  Features predicting the success of computerized decision support for prescribing: a systematic review of randomized controlled trials , 2009, BMC Medical Informatics Decis. Mak..

[8]  Oscar Mauricio Serrano Jaimes,et al.  EVALUACION DE LA USABILIDAD EN SITIOS WEB, BASADA EN EL ESTANDAR ISO 9241-11 (International Standard (1998) Ergonomic requirements For office work with visual display terminals (VDTs)-Parts II: Guidance on usability , 2012 .

[9]  Richard E. Cordes.  The effects of running fewer subjects on time-on-task measures , 1993, Int. J. Hum. Comput. Interact..

[10]  Gilad J. Kuperman,et al.  Synthesis of Research Paper: A Consensus Statement on Considerations for a Successful CPOE Implementation , 2003, J. Am. Medical Informatics Assoc..

[11]  C. Schoen,et al.  On the front lines of care: primary care doctors' office systems, experiences, and views in seven countries. , 2006, Health affairs.

[12]  P. Sedgwick.  Convenience sampling , 2013, The SAGE Encyclopedia of Research Design.

[13]  Linda Harrington,et al.  Realizing Electronic Medical Record Benefits: An Easy-to-Do Usability Study , 2011, The Journal of nursing administration.

[14]  Monique W. M. Jaspers,et al.  Effect of predefined order sets and usability problems on efficiency of computerized medication ordering , 2010, Int. J. Medical Informatics.

[15]  Maarten van Someren,et al.  The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes , 1994 .

[16]  Jeff Sauro,et al.  The Factor Structure of the System Usability Scale , 2009, HCI.

[17]  R Khajouei,et al.  The Impact of CPOE Medication Systems’ Design Aspects on Usability, Workflow and Medication Orders , 2010, Methods of Information in Medicine.

[18]  G. Pugliese,et al.  Severe Streptococcus pyogenes Infections, United Kingdom, 2003–2004 , 2008, Emerging infectious diseases.

[19]  K. Baumlin,et al.  A Pilot Study on Usability Analysis of Emergency Department Information System by Nurses , 2012, Applied Clinical Informatics.

[20]  Joseph L. Kannry,et al.  Integrating usability testing and think-aloud protocol analysis with "near-live" clinical simulations in evaluating clinical decision support , 2012, Int. J. Medical Informatics.

[21]  M. Stewart,et al.  Implementing electronic health records: Key factors in primary care. , 2008, Canadian family physician Medecin de famille canadien.

[22]  C. Goldzweig,et al.  Costs and benefits of health information technology: new trends from the literature. , 2009, Health affairs.

[23]  J. Bruhn.  Primary Care: Americaʼs Health in a New Era , 1999 .

[24]  Reinhold Haux,et al.  Primary Care Providers' Acceptance of Health Information Exchange Utilizing IHE XDS , 2013, ICIMTH.

[25]  Kathryn L. Went,et al.  Reducing prescribing errors: can a well-designed electronic system help? , 2010, Journal of evaluation in clinical practice.

[26]  J Dudeck,et al.  Evaluation of clinical information systems. What can be evaluated and what cannot? , 2001, Journal of evaluation in clinical practice.

[27]  Philip T. Kortum,et al.  Determining what individual SUS scores mean: adding an adjective rating scale , 2009 .

[28]  D. Woodwell,et al.  Physician adoption of electronic health record systems: United States, 2011. , 2012, NCHS data brief.

[29]  O. Sagher,et al.  Do Electronic Health Records Help or Hinder Medical Education? , 2009, PLoS medicine.

[30]  Michael G Leu,et al.  Effects of CPOE on Provider Cognitive Workload: A Randomized Crossover Trial , 2012, Pediatrics.

[31]  Emily S. Patterson,et al.  Identifying barriers to the effective use of clinical reminders: Bootstrapping multiple methods , 2005, J. Biomed. Informatics.

[32]  Carol M. Barnum.  The ‘magic number 5’: Is it enough for web-testing? , 2003 .

[33]  N. Menachemi,et al.  Benefits and drawbacks of electronic health record systems , 2011 .

[34]  A. Kiss,et al.  Electronic prescribing in an ambulatory care setting: a cluster randomized trial. , 2012, Journal of evaluation in clinical practice.

[35]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .