Towards Usable E-Health

BACKGROUND The use of e-health can lead to several positive outcomes. However, the potential for e-health to improve healthcare is partially dependent on its ease of use. In order to determine the usability for any technology, rigorously developed and appropriate measures must be chosen. OBJECTIVES To identify psychometrically tested questionnaires that measure usability of e-health tools, and to appraise their generalizability, attributes coverage, and quality. METHODS We conducted a systematic review of studies that measured usability of e-health tools using four databases (Scopus, PubMed, CINAHL, and HAPI). Non-primary research, studies that did not report measures, studies with children or people with cognitive limitations, and studies about assistive devices or medical equipment were systematically excluded. Two authors independently extracted information including: questionnaire name, number of questions, scoring method, item generation, and psychometrics using a data extraction tool with pre-established categories and a quality appraisal scoring table. RESULTS Using a broad search strategy, 5,558 potentially relevant papers were identified. After removing duplicates and applying exclusion criteria, 35 articles remained that used 15 unique questionnaires. From the 15 questionnaires, only 5 were general enough to be used across studies. Usability attributes covered by the questionnaires were: learnability (15), efficiency (12), and satisfaction (11). Memorability (1) was the least covered attribute. Quality appraisal showed that face/content (14) and construct (7) validity were the most frequent types of validity assessed. All questionnaires reported reliability measurement. Some questionnaires scored low in the quality appraisal for the following reasons: limited validity testing (7), small sample size (3), no reporting of user centeredness (9) or feasibility estimates of time, effort, and expense (7). CONCLUSIONS Existing questionnaires provide a foundation for research on e-health usability. However, future research is needed to broaden the coverage of the usability attributes and psychometric properties of the available questionnaires.

[1]  N. Tallent.  Psychological testing. , 1960, The American journal of nursing.

[2]  Kent L. Norman,et al.  Development of an instrument measuring user satisfaction of the human-computer interface , 1988, CHI '88.

[3]  James R. Lewis,et al.  Integrated office software benchmarks: A case study , 1990, INTERACT.

[4]  James R. Lewis,et al.  Psychometric evaluation of an after-scenario questionnaire for computer usability studies: the ASQ , 1991, SGCH.

[5]  K. Arheart,et al.  Face validity. , 1992, Western journal of nursing research.

[6]  James R. Lewis.  Psychometric Evaluation of the Post-Study System Usability Questionnaire: The PSSUQ , 1992 .

[7]  Jakob Nielsen,et al.  Chapter 6 – Usability Testing , 1993 .

[8]  R. Mcdermott,et al.  Health Education Evaluation and Measurement: A Practitioner's Perspective , 1993 .

[9]  James R. Lewis,et al.  IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use , 1995, Int. J. Hum. Comput. Interact..

[10]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[11]  D. Altman,et al.  Statistics notes: Cronbach's alpha , 1997 .

[12]  R. Kane.  Understanding Health Care Outcomes Research , 1997 .

[13]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[14]  J. Greenhalgh,et al.  Reviewing and selecting outcome measures for use in routine practice. , 1998, Journal of evaluation in clinical practice.

[15]  Timo Jokela,et al.  The standard of user-centered design and the standard definition of usability: analyzing ISO 13407 against ISO 9241-11 , 2003, CLIHC '03.

[16]  D. Streiner.  Starting at the Beginning: An Introduction to Coefficient Alpha and Internal Consistency , 2003, Journal of personality assessment.

[17]  John L.P. Thompson,et al.  Missing data , 2004, Amyotrophic lateral sclerosis and other motor neuron disorders : official publication of the World Federation of Neurology, Research Group on Motor Neuron Diseases.

[18]  J. Bloomfield,et al.  An exploration of four web-based open and flexible learning modules in post-registration nurse education. , 2004, International journal of nursing studies.

[19]  Alain Abran,et al.  Usability Meanings and Interpretations in ISO Standards , 2003, Software Quality Journal.

[20]  A. Localio,et al.  Role of computerized physician order entry systems in facilitating medication errors. , 2005, JAMA.

[21]  Paul N. Gorman,et al.  Usability testing of digital pen and paper system in nursing documentation. , 2005, AMIA ... Annual Symposium proceedings. AMIA Symposium.

[22]  V. Ostojić,et al.  Improving asthma control through telemedicine: a study of short-message service. , 2005, Telemedicine journal and e-health : the official journal of the American Telemedicine Association.

[23]  E. Im,et al.  Evaluation of the decision support computer program for cancer pain management. , 2006, Oncology nursing forum.

[24]  N. Staggers,et al.  Nurses' Evaluations of a Novel Design for an Electronic Medication Administration Record , 2007, Computers, informatics, nursing : CIN.

[25]  J. Quesada,et al.  One-year efficacy and safety of Web-based follow-up using cellular phone in type 1 diabetic patients under insulin pump therapy: the PumpNet study. , 2007, Diabetes & metabolism.

[26]  James T. Miller,et al.  An Empirical Evaluation of the System Usability Scale , 2008, Int. J. Hum. Comput. Interact..

[27]  D. Streiner,et al.  Health measurement scales , 2008 .

[28]  Ting-Ting Lee,et al.  Two-stage evaluation of the impact of a nursing information system in Taiwan , 2008, Int. J. Medical Informatics.

[29]  R. Darlington,et al.  Factor Analysis , 2008 .

[30]  Jeff Yan,et al.  Usability of CAPTCHAs or usability issues in CAPTCHA design , 2008, SOUPS '08.

[31]  Herbert S. Lin,et al.  Computational Technology for Effective Health Care: Immediate Steps and Strategic Directions , 2009 .

[32]  Maria E Fernandez,et al.  How we design feasibility studies. , 2009, American journal of preventive medicine.

[33]  Jeff Sauro,et al.  The Factor Structure of the System Usability Scale , 2009, HCI.

[34]  Carolyn A. Lin,et al.  Usability Testing by Older Adults of a Computer-Mediated Health Communication Program , 2009, Journal of health communication.

[35]  Monique W. M. Jaspers,et al.  A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence , 2009, Int. J. Medical Informatics.

[36]  M. Dew,et al.  Evaluation of a hand‐held, computer‐based intervention to promote early self‐care behaviors after lung transplant , 2009, Clinical transplantation.

[37]  S. Bakken,et al.  Development of a Customizable Health IT Usability Evaluation Scale. , 2010, AMIA ... Annual Symposium proceedings. AMIA Symposium.

[38]  S. Salanterä,et al.  The Creation and Evaluation of an Ambulatory Orthopedic Surgical Patient Education Web Site to Support Empowerment , 2010, Computers, informatics, nursing : CIN.

[39]  W. Jack,et al.  Effects of a mobile phone short message service on antiretroviral treatment adherence in Kenya (WelTel Kenya1): a randomised trial , 2010, The Lancet.

[40]  John Mark Ansermino,et al.  A human factors and survey methodology-based design of a web-based adverse event reporting system for families , 2010, Int. J. Medical Informatics.

[41]  Asil Oztekin,et al.  UseLearn: A novel checklist and usability evaluation method for eLearning systems by criticality metric analysis , 2010 .

[42]  Richard J. Holden,et al.  The Technology Acceptance Model: Its past and its future in health care , 2010, J. Biomed. Informatics.

[43]  Farzad Mostashari,et al.  Using electronic health record alerts to provide public health situational awareness to clinicians , 2010, J. Am. Medical Informatics Assoc..

[44]  Karen Holtzblatt.  Lifetime Practice Award , 2010, CHI 2010.

[45]  Jiajie Zhang,et al.  TURF: Toward a unified framework of EHR usability , 2011, J. Biomed. Informatics.

[46]  M. Lim,et al.  Impact of text and email messaging on the sexual health of young people: a randomised controlled trial , 2011, Journal of Epidemiology & Community Health.

[47]  Jason J. Saleem,et al.  Redesign of a computerized clinical reminder for colorectal cancer screening: a human-computer interaction evaluation , 2011, BMC Medical Informatics Decis. Mak..

[48]  Torgny Groth,et al.  A web application for follow-up of results from a mobile device test battery for Parkinson's disease patients , 2011, Comput. Methods Programs Biomed..

[49]  Ting-Ting Lee,et al.  Evaluation of ICU Nurses' Use of the Clinical Information System in Taiwan , 2011, Computers, informatics, nursing : CIN.

[50]  S. Pongpanich,et al.  Tuberculosis treatment with mobile-phone medication reminders in northern Thailand. , 2011, The Southeast Asian journal of tropical medicine and public health.

[51]  S. Stjernswärd,et al.  Illuminating User Experience of a Website for the Relatives of Persons With Depression , 2011, The International journal of social psychiatry.

[52]  Peter Hoonakker,et al.  ICU nurses' acceptance of electronic health records , 2011, J. Am. Medical Informatics Assoc..

[53]  W. Stone,et al.  A web-based tutorial for parents of young children with autism: results from a pilot study. , 2011, Telemedicine journal and e-health : the official journal of the American Telemedicine Association.

[54]  K. H. Gulkesen,et al.  Usability of a web-based personal nutrition management tool , 2011, Informatics for health & social care.

[55]  A. Gafni,et al.  A usability study of a computerized decision aid to help patients with, early stage papillary thyroid carcinoma in, decision-making on adjuvant radioactive iodine treatment. , 2011, Patient education and counseling.

[56]  V. Herasevich,et al.  The effect of two different electronic health record user interfaces on intensive care provider task load, errors of cognition, and performance* , 2011, Critical care medicine.

[57]  J. West,et al.  There ’ s an App for That : Content Analysis of Paid Health and Fitness Apps , 2018 .

[58]  K. Baumlin,et al.  A Pilot Study on Usability Analysis of Emergency Department Information System by Nurses , 2012, Applied Clinical Informatics.

[59]  S. Zaidi,et al.  Barriers and Facilitators to Adoption of a Web-based Antibiotic Decision Support System , 2012, Southern med review.

[60]  Charlene R. Weir,et al.  Intensive care unit nurses' information needs and recommendations for integrated displays to improve nurses' situation awareness , 2012, J. Am. Medical Informatics Assoc..

[61]  José Creissac Campos,et al.  A literature review about usability evaluation methods for e-learning platforms. , 2012, Work.

[62]  Fleur Fritz,et al.  Qualitative and quantitative evaluation of EHR-integrated mobile patient questionnaires regarding usability and cost-efficiency , 2012, Int. J. Medical Informatics.

[63]  Bey-Hwa Yui,et al.  Evaluation of Computerized Physician Order Entry System—A Satisfaction Survey in Taiwan , 2012, Journal of Medical Systems.

[64]  Carol S. Parke.  Essential First Steps to Data Analysis: Scenario-Based Examples Using SPSS , 2012 .

[65]  Anne Miller,et al.  Evaluation of an integrated graphical display to promote acute change detection in ICU patients , 2012, Int. J. Medical Informatics.

[66]  J. C. Huber,et al.  Factors affecting acceptability and usability of technological approaches to diabetes self-management: a case study. , 2012, Diabetes technology & therapeutics.

[67]  Bambang Parmanto,et al.  Evaluation of a Telerehabilitation System for Community-Based Rehabilitation , 2012, International journal of telerehabilitation.

[68]  D. Vidrine,et al.  Efficacy of cell phone-delivered smoking cessation counseling for persons living with HIV/AIDS: 3-month outcomes. , 2012, Nicotine & tobacco research : official journal of the Society for Research on Nicotine and Tobacco.

[69]  Suzanne Bakken,et al.  Review of health information technology usability study methodologies , 2011, J. Am. Medical Informatics Assoc..

[70]  Marcelo Gitirana Gomes Ferreira,et al.  Developing an usability test to evaluate the use of augmented reality to improve the first interaction with a product. , 2012, Work.

[71]  Dara Meldrum,et al.  Virtual reality rehabilitation of balance: assessment of the usability of the Nintendo Wii® Fit Plus , 2012, Disability and rehabilitation. Assistive technology.

[72]  Suzanne Bakken,et al.  Development of a prototype continuity of care record with context-specific links to meet the information needs of case managers for persons living with HIV , 2012, Int. J. Medical Informatics.

[73]  A. Mckibbon,et al.  A usability study of patients setting up a cardiac event loop recorder and BlackBerry gateway for remote monitoring at home. , 2012, Telemedicine journal and e-health : the official journal of the American Telemedicine Association.

[74]  C. Parkin,et al.  2-Year follow-up to STeP trial shows sustainability of structured self-monitoring of blood glucose utilization: results from the STeP practice logistics and usability survey (STeP PLUS). , 2013, Diabetes technology & therapeutics.

[75]  Emilie K. Johnson,et al.  Values and pitfalls of the use of administrative databases for outcomes assessment. , 2013, The Journal of urology.

[76]  Tarika Daftary-Kapur,et al.  Measuring Success: A Guide to becoming an evidence-based practice , 2013 .

[77]  Andreas Jonsson.  Usability in three generations business support systems - Assessing perceived usability in the banking industry , 2013 .

[78]  J. B. Brooke,et al.  SUS: a retrospective , 2013 .

[79]  Allen McKiel.  Academic Libraries After Print , 2013 .

[80]  R. Koppel.  Is Healthcare Information Technology Based on Evidence? , 2013, Yearbook of Medical Informatics.

[81]  A. Haines,et al.  The Effectiveness of Mobile-Health Technology-Based Health Behaviour Change or Disease Management Interventions for Health Care Consumers: A Systematic Review , 2013, PLoS medicine.

[82]  Ajit Kumar,et al.  Nursing process decision support system for urology ward , 2013, Int. J. Medical Informatics.

[83]  Diane Gromala,et al.  Usability testing of ANSWER: a web-based methotrexate decision aid for patients with rheumatoid arthritis , 2013, BMC Medical Informatics and Decision Making.

[84]  Nese Zayim,et al.  A web-based intensive care clinical decision support system: From design to evaluation , 2013, Informatics for health & social care.

[85]  H. Wollersheim,et al.  Measuring a caring culture in hospitals: a systematic review of instruments , 2013, BMJ Open.

[86]  Michael E. Levin,et al.  Feasibility of a Prototype Web-Based Acceptance and Commitment Therapy Prevention Program for College Students , 2014, Journal of American college health : J of ACH.

[87]  Wyeth W. Wasserman,et al.  Usability study of clinical exome analysis software: Top lessons learned and recommendations , 2014, J. Biomed. Informatics.

[88]  A. Costello,et al.  Development, feasibility and usability of an online psychological intervention for men with prostate cancer: My Road Ahead , 2014 .

[89]  Suzanne Bakken,et al.  Examining construct and predictive validity of the Health-IT Usability Evaluation Scale: confirmatory factor analysis and structural equation modeling results. , 2014, Journal of the American Medical Informatics Association : JAMIA.

[90]  D. Brooks,et al.  Developing a physiotherapy-specific preliminary clinical decision-making tool for oxygen titration: a modified delphi study. , 2014, Physiotherapy Canada. Physiotherapie Canada.

[91]  N. Shah,et al.  Patients Welcome the Sickle Cell Disease Mobile Application to Record Symptoms via Technology (SMART) , 2014, Hemoglobin.

[92]  A. Joshi,et al.  Evaluation of an interactive surveillance system for monitoring acute bacterial infections in Nigeria. , 2014, Perspectives in health information management.

[93]  Jiajie Zhang,et al.  The Current Status of Usability Studies of Information Technologies in China: A Systematic Study , 2014, BioMed research international.

[94]  M. Moattari,et al.  Validating a Web-based Diabetes Education Program in continuing nursing education: knowledge and competency change and user perceptions on usability and quality , 2014, Journal of Diabetes & Metabolic Disorders.

[95]  P. Pronovost,et al.  Usability and perceived usefulness of Personal Health Records for preventive health care: a case study focusing on patients' and primary care providers' perspectives. , 2014, Applied ergonomics.

[96]  L. D. de Witte,et al.  Usability testing of a monitoring and feedback tool to stimulate physical activity , 2014, Patient preference and adherence.

[97]  R. Hovorka,et al.  Glucose control in intensive care: usability, efficacy and safety of Space GlucoseControl in two medical European intensive care units , 2014, BMC Endocrine Disorders.

[98]  Aldo von Wangenheim,et al.  Does the use of structured reporting improve usability? A comparative evaluation of the usability of two approaches for findings reporting in a large-scale telecardiology context , 2014, J. Biomed. Informatics.

[99]  C. Lyles,et al.  Getting a Technology-Based Diabetes Intervention Ready for Prime Time: a Review of Usability Testing Studies , 2014, Current Diabetes Reports.

[100]  Tiago Taveira-Gomes,et al.  A novel collaborative e-learning platform for medical students - ALERT STUDENT , 2014, BMC medical education.

[101]  Bo Hsiao,et al.  The effect of user's perceived presence and promotion focus on usability for interacting in virtual environments. , 2015, Applied ergonomics.

[102]  Alessandro Febretti,et al.  Use of Simulation to Study Nurses’ Acceptance and Nonacceptance of Clinical Decision Support Suggestions , 2015, Computers, informatics, nursing : CIN.

[103]  Eric J. Topol,et al.  The emerging field of mobile health , 2015, Science Translational Medicine.

[104]  Ming-I Brandon Lin,et al.  Usage Position and Virtual Keyboard Design Affect Upper-Body Kinematics, Discomfort, and Usability during Prolonged Tablet Typing , 2015, PloS one.

[105]  Ole K. Hejlesen,et al.  Heuristic evaluation of a telehealth system from the Danish TeleCare North Trial , 2015, Int. J. Medical Informatics.

[106]  Maha Aljohani,et al.  Heuristic Evaluation of University Institutional Repositories Based on DSpace , 2015, HCI.

[107]  Carmen L. Lewis,et al.  Development and usability testing of a Web-based decision aid for families of patients receiving prolonged mechanical ventilation , 2015, Annals of Intensive Care.

[108]  Hyeoun-Ae Park,et al.  Development of a Smartphone Application for Clinical-Guideline-Based Obesity Management , 2015, Healthcare informatics research.

[109]  D. Kavanagh,et al.  The acceptability, usability and short-term outcomes of Get Real: A web-based program for psychotic-like experiences (PLEs) , 2015 .

[110]  J. Glanz,et al.  Development of an Interactive Social Media Tool for Parents With Concerns About Vaccines , 2015, Health education & behavior : the official publication of the Society for Public Health Education.

[111]  M. Shah,et al.  The impacts of second generation e-prescribing usability on community pharmacists outcomes. , 2015, Research in social & administrative pharmacy : RSAP.

[112]  Samantha R. Paige,et al.  Engaging Community Stakeholders to Evaluate the Design, Usability, and Acceptability of a Chronic Obstructive Pulmonary Disease Social Media Resource Center , 2015, JMIR research protocols.

[113]  John T. Wei,et al.  Evaluation of point-of-care PRO assessment in clinic settings: integration, parallel-forms reliability, and patient acceptability of electronic QOL measures during clinic visits , 2016, Quality of Life Research.

[114]  Joel J. P. C. Rodrigues,et al.  Mobile-health: A review of current state in 2015 , 2015, J. Biomed. Informatics.

[115]  L. Atack,et al.  Simulation and gaming to promote health education: Results of a usability test , 2015 .

[116]  A. Bauman,et al.  A Study to Determine the Most Popular Lifestyle Smartphone Applications and Willingness of the Public to Share Their Personal Data for Health Research. , 2016, Telemedicine journal and e-health : the official journal of the American Telemedicine Association.

[117]  H. Seidling,et al.  Best practice strategies to safeguard drug prescribing and drug administration: an anthology of expert views and opinions , 2016, International Journal of Clinical Pharmacy.

[118]  Juho Tapani.  Game usability in North American video game industry , 2016 .

[119]  W. Groot,et al.  The use of e-health and m-health tools in health promotion and primary prevention among older adults: a systematic literature review , 2016, BMC Health Services Research.

[120]  R. Ratwani,et al.  Barriers to comparing the usability of electronic health records , 2017, J. Am. Medical Informatics Assoc..

[121]  V. Sousa,et al.  Questionnaire Adapting: Little Changes Mean a Lot , 2017, Western journal of nursing research.

[122]  Alessandro Febretti,et al.  Toward a More Robust and Efficient Usability Testing Method of Clinical Decision Support for Nurses Derived From Nursing Electronic Health Record Data. , 2017, International journal of nursing knowledge.

[123]  R. Chen,et al.  Examining Usability of E-learning Systems-An Exploratory Study ( Research-in-Progress ) , 2022 .