Does the use of structured reporting improve usability? A comparative evaluation of the usability of two approaches for findings reporting in a large-scale telecardiology context

One of the main reasons that leads to a low adoption rate of telemedicine systems is poor usability. An aspect that influences usability during the reporting of findings is the input mode, e.g., if a free-text (FT) or a structured report (SR) interface is employed. The objective of our study is to compare the usability of FT and ST telemedicine systems, specifically in terms of user satisfaction, efficiency and general usability. We comparatively evaluate the usability of these two input modes in a telecardiology system for issuing electrocardiography reports in the context of a statewide telemedicine system in Brazil with more than 350.000 performed tele-electrocardiography examinations. We adopted a multiple method research strategy, applying three different kinds of usability evaluations: user satisfaction was evaluated through interviews with seven medical professionals using the System Usability Scale (SUS) questionnaire and specific questions related to adequacy and user experience. Efficiency was evaluated by estimating execution time using the Keystroke-Level Model (KLM). General usability was assessed based on the conformity of the systems to a set of e-health specific usability heuristics. The results of this comparison provide a first indication that a structured report (SR) input mode for such a system is more satisfactory and efficient with a larger conformity to usability heuristics than free-text (FT) input. User satisfaction using the SUS questionnaire has been scored in average with 58.8 and 77.5 points for the FT and SR system, respectively, which means that the SR system was rated 18.65 points higher than the FT system. In terms of efficiency, the completion of a findings report using the SR mode is estimated to take 8.5s, 3.74 times faster than using the FT system (31.8s). The SR system also demonstrated less violations to usability heuristics (8 points) in comparison to 14 points observed in the FT system. These results provide a first indication that the usage of structured reporting as an input mode in telecardiology systems may enhance usability. This also seems to confirm the advantages of the usage of structured reporting, as already described in the literature for other areas such as teleradiology.

[1]  Gavriel Salvendy,et al.  Handbook of Human Factors and Ergonomics , 2005 .

[2]  P. Parizel,et al.  Structured reporting: if, why, when, how—and at what expense? Results of a focus group meeting of radiology professionals from eight countries , 2012, Insights into Imaging.

[3]  Angela M. Cirucci,et al.  Usability Testing , 2021, UX Research Methods for Media and Communication Studies.

[4]  Jakob Nielsen,et al.  Measuring usability: preference vs. performance , 1994, CACM.

[5]  Lucy Suchman.  Plans and situated actions: the problem of human-machine communication , 1987 .

[6]  Oscar Mauricio Serrano Jaimes,et al.  EVALUACION DE LA USABILIDAD EN SITIOS WEB, BASADA EN EL ESTANDAR ISO 9241-11 (International Standard (1998) Ergonomic requirements For office work with visual display terminals (VDTs)-Parts II: Guidance on usability , 2012 .

[7]  Emily S. Patterson,et al.  Technical evaluation, testing, and validatiaon of the usability of electronic health records , 2012 .

[8]  Austin Henderson,et al.  Interaction design: beyond human-computer interaction , 2002, UBIQ.

[9]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[10]  E. McGlynn,et al.  The Challenge of Measuring Quality of Care From the Electronic Health Record , 2009, American journal of medical quality : the official journal of the American College of Medical Quality.

[11]  Linn Wikström,et al.  An application design supporting structured radiology reports , 2007 .

[12]  Adinda Freudenthal,et al.  Design for risk control: The role of usability engineering in the management of use-related risks , 2012, J. Biomed. Informatics.

[13]  Patrick W. Jordan,et al.  An Introduction to Usability , 1998 .

[14]  Blackford Middleton,et al.  Barriers to Electronic Health Record Use during Patient Visits , 2006, AMIA.

[15]  Aldo von Wangenheim,et al.  Building a National Telemedicine Network , 2008, IT Professional.

[16]  Allen Newell,et al.  The psychology of human-computer interaction , 1983 .

[17]  K. Nikus,et al.  The Role of Standard 12-lead ECG in a Telecardiology Consultation Service , 2011 .

[18]  James T. Miller,et al.  An Empirical Evaluation of the System Usability Scale , 2008, Int. J. Hum. Comput. Interact..

[19]  Y. Rogers,et al.  Interaction Design , 2002 .

[20]  David L Weiss,et al.  Structured reporting: patient care enhancement or productivity nightmare? , 2008, Radiology.

[21]  Electronic Health Record Usability Interface Design Considerations , 2009 .

[22]  William Brown,et al.  Assessment of the Health IT Usability Evaluation Model (Health-ITUEM) for evaluating mobile health (mHealth) technology , 2013, J. Biomed. Informatics.

[23]  Christian Nøhr,et al.  Development of Methods for Usability Evaluations of EHR Systems , 2006, MIE.

[24]  Vimla L. Patel,et al.  Usability testing in medical informatics: cognitive approaches to evaluation of information systems and user interfaces , 1997, AMIA.

[25]  Curtis P Langlotz,et al.  Structured radiology reporting: are we there yet? , 2009, Radiology.

[26]  Valdair Francisco Muglia,et al.  Radiological Reports: A Comparison between the Transmission Efficiency of Information in Free Text and in Structured Reports , 2010, Clinics.

[27]  Silvia Modesto Nassar,et al.  User satisfaction with asynchronous telemedicine: a study of users of Santa Catarina's system of telemedicine and telehealth. , 2012, Telemedicine journal and e-health : the official journal of the American Telemedicine Association.

[28]  R. Wootton,et al.  Introduction to Telemedicine , 1999 .

[29]  D. Blumenthal.  Stimulating the adoption of health information technology. , 2009, The West Virginia medical journal.

[30]  Jos Aarts,et al.  Human factors engineering for healthcare IT clinical applications , 2010, Int. J. Medical Informatics.

[31]  D. Gans,et al.  Medical groups' adoption of electronic health records and information systems. , 2005, Health affairs.

[32]  Sowmya R. Rao,et al.  Electronic health records in ambulatory care--a national survey of physicians. , 2008, The New England journal of medicine.

[33]  Sri Hastuti Kurniawan,et al.  Review of Interaction design , 2003 .

[34]  J. B. Brooke,et al.  SUS: a retrospective , 2013 .

[35]  G. Bartoo,et al.  Essentials of usability engineering in point-of-care devices , 2013, 2013 IEEE Point-of-Care Healthcare Technologies (PHT).

[36]  Jiajie Zhang,et al.  TURF: Toward a unified framework of EHR usability , 2011, J. Biomed. Informatics.

[37]  S. Soderland,et al.  Automatic structuring of radiology free-text reports. , 2001, Radiographics : a review publication of the Radiological Society of North America, Inc.

[38]  Yvonne Rogers,et al.  Interaction Design: Beyond Human-Computer Interaction , 2002 .