Are three methods better than one? A comparative assessment of usability evaluation methods in an EHR

OBJECTIVE To comparatively evaluate the effectiveness of three different methods involving end-users for detecting usability problems in an EHR: user testing, semi-structured interviews and surveys. MATERIALS AND METHODS Data were collected at two major urban dental schools from faculty, residents and dental students to assess the usability of a dental EHR for developing a treatment plan. These included user testing (N=32), semi-structured interviews (N=36), and surveys (N=35). RESULTS The three methods together identified a total of 187 usability violations: 54% via user testing, 28% via the semi-structured interview and 18% from the survey method, with modest overlap. These usability problems were classified into 24 problem themes in 3 broad categories. User testing covered the broadest range of themes (83%), followed by the interview (63%) and survey (29%) methods. DISCUSSION Multiple evaluation methods provide a comprehensive approach to identifying EHR usability challenges and specific problems. The three methods were found to be complementary, and thus each can provide unique insights for software enhancement. Interview and survey methods were found not to be sufficient by themselves, but when used in conjunction with the user testing method, they provided a comprehensive evaluation of the EHR. CONCLUSION We recommend using a multi-method approach when testing the usability of health information technology because it provides a more comprehensive picture of usability challenges.

[1]  Adam Langley,et al.  The Use of Information Technologies Among Rural and Urban Physicians in Florida , 2007, Journal of Medical Systems.

[2]  Vimla L. Patel,et al.  To err is not entirely human: Complex technology and user cognition , 2005, J. Biomed. Informatics.

[3]  Reinhold Haux,et al.  Health information systems - past, present, future , 2006, Int. J. Medical Informatics.

[4]  Robin C. Meili,et al.  Can electronic medical record systems transform health care? Potential health benefits, savings, and costs. , 2005, Health affairs.

[5]  Kenneth R. Koedinger,et al.  Predictive human performance modeling made easy , 2004, CHI.

[6]  Dongwen Wang,et al.  Using multi-perspective methodologies to study users' interactions with the prototype front end of a guideline-based decision support system for diabetic foot care , 2009, Int. J. Medical Informatics.

[7]  Tinja Lääveri,et al.  National questionnaire study on clinical ICT systems proofs: Physicians suffer from poor usability , 2011, Int. J. Medical Informatics.

[8]  Anastasia Papazafeiropoulou,et al.  Investigating evaluation frameworks for health information systems , 2008, Int. J. Medical Informatics.

[9]  Jiajie Zhang,et al.  TURF: Toward a unified framework of EHR usability , 2011, J. Biomed. Informatics.

[10]  Jiajie Zhang,et al.  Human-centered computing in health information systems Part 1: Analysis and design , 2005, J. Biomed. Informatics.

[11]  Monique W. M. Jaspers,et al.  The significance of a usability evaluation of an emerging laboratory order entry system , 2007, Int. J. Medical Informatics.

[12]  David W. Bates,et al.  Reducing the frequency of errors in medicine using information technology. , 2001, Journal of the American Medical Informatics Association : JAMIA.

[13]  A. Localio,et al.  Role of computerized physician order entry systems in facilitating medication errors. , 2005 .

[14]  Allen Newell,et al.  The psychology of human-computer interaction , 1983 .

[15]  Y. Han,et al.  Unexpected Increased Mortality After Implementation of a Commercially Sold Computerized Physician Order Entry System , 2005, Pediatrics.

[16]  Titus Schleyer,et al.  A usability evaluation of four commercial dental computer-based patient record systems. , 2008, Journal of the American Dental Association.

[17]  Gilad J. Kuperman,et al.  Case Report: Comprehensive Analysis of a Medication Dosing Error Related to CPOE , 2005, J. Am. Medical Informatics Assoc..

[18]  Vimla L. Patel,et al.  Cognitive and usability engineering methods for the evaluation of clinical information systems , 2004, J. Biomed. Informatics.

[19]  John Doucette,et al.  Primary Care Physicians' Experience with Electronic Medical Records: Barriers to Implementation in a Fee-for-Service Environment , 2008, International journal of telemedicine and applications.

[20]  Elizabeth M Borycki,et al.  Technology induced error and usability: The relationship between usability problems and prescription errors when using a handheld application , 2005, Int. J. Medical Informatics.

[21]  Sarah J Clark,et al.  Adoption of Electronic Health Records in Primary Care Pediatric Practices , 2006, Pediatrics.

[22]  Vimla L. Patel,et al.  Assessing performance of an Electronic Health Record (EHR) using Cognitive Task Analysis , 2010, Int. J. Medical Informatics.

[23]  Kaveh G Shojania,et al.  Usability evaluation of order sets in a computerised provider order entry system , 2011, BMJ quality & safety.

[24]  R. Khajouei,et al.  Classification and prioritization of usability problems using an augmented classification scheme , 2011, J. Biomed. Informatics.

[25]  Emily S. Patterson,et al.  (NISTIR 7804) Technical Evaluation, Testing and Validation of the Usability of Electronic Health Records , 2012 .

[26]  Qi Li,et al.  Using qualitative studies to improve the usability of an EMR , 2005, J. Biomed. Informatics.

[27]  D. Blumenthal,et al.  Information technology comes to medicine. , 2007, The New England journal of medicine.

[28]  P. Shekelle,et al.  Systematic Review: Impact of Health Information Technology on Quality, Efficiency, and Costs of Medical Care , 2006, Annals of Internal Medicine.

[29]  D. Chou.  Health IT and Patient Safety: Building Safer Systems for Better Care , 2012 .

[30]  Emily S. Patterson,et al.  Technical evaluation, testing, and validatiaon of the usability of electronic health records , 2012 .

[31]  Joan S. Ash,et al.  Research Paper: Types of Unintended Consequences Related to Computerized Provider Order Entry , 2006, J. Am. Medical Informatics Assoc..

[32]  Joan S Ash,et al.  Health Information Technology Systems profoundly impact users: a case study in a dental school. , 2010, Journal of dental education.

[33]  Jiajie Zhang,et al.  Can prospective usability evaluation predict data errors? , 2010, AMIA ... Annual Symposium proceedings. AMIA Symposium.

[34]  Nicolette de Keizer,et al.  Guideline for good evaluation practice in health informatics (GEP-HI) , 2011, Int. J. Medical Informatics.

[35]  Marc Berg,et al.  Viewpoint Paper: Some Unintended Consequences of Information Technology in Health Care: The Nature of Patient Care Information System-related Errors , 2003, J. Am. Medical Informatics Assoc..

[36]  Monique W. M. Jaspers,et al.  A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence , 2009, Int. J. Medical Informatics.

[37]  H. Lærum,et al.  Doctors' use of electronic medical records systems in hospitals: cross sectional survey , 2001, BMJ : British Medical Journal.

[38]  L. Petersen,et al.  Barriers to proliferation of electronic medical records. , 2004, Informatics in primary care.

[39]  Blackford Middleton,et al.  Complementary methods of system usability evaluation: Surveys and observations during software design and development cycles , 2010, J. Biomed. Informatics.

[40]  G. A. Loomis,et al.  If electronic medical records are so great, why aren't family physicians using them? , 2002, The Journal of family practice.

[41]  Monique W. M. Jaspers,et al.  Determination of the effectiveness of two methods for usability evaluation using a CPOE medication ordering system , 2011, Int. J. Medical Informatics.

[42]  Charlotte A. Weaver,et al.  Enhancing patient safety and quality of care by improving the usability of electronic health record systems: recommendations from AMIA. , 2013, Journal of the American Medical Informatics Association : JAMIA.

[43]  Monique W. M. Jaspers,et al.  Effect of predefined order sets and usability problems on efficiency of computerized medication ordering , 2010, Int. J. Medical Informatics.

[44]  Vickie Nguyen,et al.  Detection and characterization of usability problems in structured data entry interfaces in dentistry , 2013, Int. J. Medical Informatics.

[45]  Joel M. White,et al.  The development of a dental diagnostic terminology. , 2011, Journal of dental education.

[46]  Vimla L. Patel,et al.  Translational cognition for decision support in critical care environments: A review , 2008, J. Biomed. Informatics.

[47]  Jiajie Zhang,et al.  Human-centered computing in health information systems: Part 2: Evaluation , 2005, J. Biomed. Informatics.