Assessment of the Health IT Usability Evaluation Model (Health-ITUEM) for evaluating mobile health (mHealth) technology

BACKGROUND Over two decades of research has been conducted using mobile devices for health related behaviors yet many of these studies lack rigor. There are few evaluation frameworks for assessing the usability of mHealth, which is critical as the use of this technology proliferates. As the development of interventions using mobile technology increase, future work in this domain necessitates the use of a rigorous usability evaluation framework. METHODS We used two exemplars to assess the appropriateness of the Health IT Usability Evaluation Model (Health-ITUEM) for evaluating the usability of mHealth technology. In the first exemplar, we conducted 6 focus group sessions to explore adolescents' use of mobile technology for meeting their health Information needs. In the second exemplar, we conducted 4 focus group sessions following an Ecological Momentary Assessment study in which 60 adolescents were given a smartphone with pre-installed health-related applications (apps). DATA ANALYSIS We coded the focus group data using the 9 concepts of the Health-ITUEM: Error prevention, Completeness, Memorability, Information needs, Flexibility/Customizability, Learnability, Performance speed, Competency, Other outcomes. To develop a finer granularity of analysis, the nine concepts were broken into positive, negative, and neutral codes. A total of 27 codes were created. Two raters (R1 and R2) initially coded all text and a third rater (R3) reconciled coding discordance between raters R1 and R2. RESULTS A total of 133 codes were applied to Exemplar 1. In Exemplar 2 there were a total of 286 codes applied to 195 excerpts. Performance speed, Other outcomes, and Information needs were among the most frequently occurring codes. CONCLUSION Our two exemplars demonstrated the appropriateness and usefulness of the Health-ITUEM in evaluating mobile health technology. Further assessment of this framework with other study populations should consider whether Memorability and Error prevention are necessary to include when evaluating mHealth technology.

[1]  Suzanne Bakken,et al.  Employing the FITT framework to explore HIV case managers' perceptions of two electronic clinical data (ECD) summary systems , 2012, Int. J. Medical Informatics.

[2]  J. Creswell.  Qualitative inquiry and research design: choosing among five traditions. , 1998 .

[3]  Eli Lieber,et al.  Mixing Qualitative and Quantitative Methods: Insights into Design and Analysis Issues. , 2009 .

[4]  J. Wolf,et al.  Diagnostic inaccuracy of smartphone applications for melanoma detection. , 2013, JAMA dermatology.

[5]  Oscar Mauricio Serrano Jaimes,et al.  EVALUACION DE LA USABILIDAD EN SITIOS WEB, BASADA EN EL ESTANDAR ISO 9241-11 (International Standard (1998) Ergonomic requirements For office work with visual display terminals (VDTs)-Parts II: Guidance on usability , 2012 .

[6]  Jan Bosch,et al.  Architecting for usability: a survey , 2004, J. Syst. Softw..

[7]  Tiago Jacinto,et al.  Challenges of a mobile application for asthma and allergic rhinitis patient enablement-interface and synchronization. , 2013, Telemedicine journal and e-health : the official journal of the American Telemedicine Association.

[8]  J. Car,et al.  Mobile phone messaging for preventive health care. , 2012, The Cochrane database of systematic reviews.

[9]  M. Angen.  Evaluating Interpretive Inquiry: Reviewing the Validity Debate and Opening the Dialogue , 2000, Qualitative health research.

[10]  Po-Yin Yen.  Health information technology usability evaluation: methods, models, and measures , 2010 .

[11]  Vincent Mysliwiec,et al.  Usability and feasibility of smartphone video capabilities for telehealth care in the U.S. military. , 2012, Telemedicine journal and e-health : the official journal of the American Telemedicine Association.

[12]  A. Haines,et al.  The Effectiveness of Mobile-Health Technology-Based Health Behaviour Change or Disease Management Interventions for Health Care Consumers: A Systematic Review , 2013, PLoS medicine.

[13]  Ben Shneiderman,et al.  Designing the User Interface: Strategies for Effective Human-Computer Interaction , 1998 .

[14]  Jakob Nielsen,et al.  Usability inspection methods , 1994, CHI 95 Conference Companion.

[15]  Ilkka Korhonen,et al.  Mobile Diary for Wellness Management—Results on Usage and Usability in Two User Studies , 2008, IEEE Transactions on Information Technology in Biomedicine.

[16]  B Sheehan,et al.  A comparison of usability factors of four mobile devices for accessing healthcare information by adolescents. , 2012, Applied clinical informatics.

[17]  Per Carlbring,et al.  Directions for the International Society for Research on Internet Interventions (ISRII) , 2006, Journal of medical Internet research.

[18]  Ralph Maddison,et al.  A Development and Evaluation Process for mHealth Interventions: Examples From New Zealand , 2012, Journal of health communication.

[19]  K. Patrick,et al.  Health and the mobile phone. , 2008, American journal of preventive medicine.

[20]  D. Wantland,et al.  The Effectiveness of Web-Based vs. Non-Web-Based Interventions: A Meta-Analysis of Behavioral Change Outcomes , 2004, Journal of medical Internet research.

[21]  L. Abroms,et al.  Text2Quit: Results From a Pilot Test of a Personalized, Interactive Mobile Health Smoking Cessation Program , 2012, Journal of health communication.

[22]  Fred D. Davis.  Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology , 1989, MIS Q..

[23]  M. Sandelowski.  Rigor or rigor mortis: the problem of rigor in qualitative research revisited. , 1993, ANS. Advances in nursing science.

[24]  Rebecca Schnall,et al.  Using Text Messaging to Assess Adolescents' Health Information Needs: An Ecological Momentary Assessment , 2013, Journal of medical Internet research.

[25]  Suzanne Bakken,et al.  Applying an Evaluation Framework for Health Information System Design, Development, and Implementation , 2006, Nursing research.

[26]  J. Cimino.  Desiderata for Controlled Medical Vocabularies in the Twenty-First Century , 1998, Methods of Information in Medicine.

[27]  Joan S. Ash,et al.  Qualitative evaluation of health information exchange efforts , 2007, J. Biomed. Informatics.

[28]  Ralph Maddison,et al.  A mHealth cardiac rehabilitation exercise intervention: findings from content development studies , 2012, BMC Cardiovascular Disorders.

[29]  Joseph F. Dumas,et al.  A Practical Guide to Usability Testing , 1993 .

[30]  S. Kossman,et al.  Nurses' Perceptions of the Impact of Electronic Health Records on Work and Patient Outcomes , 2008, Computers, informatics, nursing : CIN.

[31]  Alain Abran,et al.  Usability Meanings and Interpretations in ISO Standards , 2003, Software Quality Journal.

[32]  F. Griffiths,et al.  Why Are Health Care Interventions Delivered Over the Internet? A Systematic Review of the Published Literature , 2006, Journal of medical Internet research.

[33]  Joe H. Brown,et al.  Back to basics: Percentage agreement measures are adequate, but there are easier ways. , 1979, Journal of applied behavior analysis.

[34]  Colin Potts,et al.  Design of Everyday Things , 1988 .

[35]  Dongsong Zhang,et al.  Challenges, Methodologies, and Issues in the Usability Testing of Mobile Applications , 2005, Int. J. Hum. Comput. Interact..

[36]  C. Sherbourne,et al.  The MOS 36-item Short-Form Health Survey (SF-36): III. Tests of data quality, scaling assumptions, and reliability across diverse patient groups. , 1994 .

[37]  Jakob Nielsen,et al.  Heuristic Evaluation of Prototypes (individual) , 2022 .

[38]  A. Mckibbon,et al.  A usability study of patients setting up a cardiac event loop recorder and BlackBerry gateway for remote monitoring at home. , 2012, Telemedicine journal and e-health : the official journal of the American Telemedicine Association.

[39]  Emily S. Patterson,et al.  Research Paper: Improving Patient Safety by Identifying Side Effects from Introducing Bar Coding in Medication Administration , 2002, J. Am. Medical Informatics Assoc..

[40]  Joseph L. Kannry,et al.  The Relationship of Usability to Medical Error: An Evaluation of Errors Associated with Usability Problems in the Use of a Handheld Application for Prescribing Medications , 2004, MedInfo.