Methods for Evaluating the Content, Usability, and Efficacy of Commercial Mobile Health Apps

Commercial mobile apps for health behavior change are flourishing in the marketplace, but little evidence exists to support their use. This paper summarizes methods for evaluating the content, usability, and efficacy of commercially available health apps. Content analyses can be used to compare app features with clinical guidelines, evidence-based protocols, and behavior change techniques. Usability testing can establish how well an app functions and serves its intended purpose for a target population. Observational studies can explore the association between use and clinical and behavioral outcomes. Finally, efficacy testing can establish whether a commercial app impacts an outcome of interest via a variety of study designs, including randomized trials, multiphase optimization studies, and N-of-1 studies. Evidence in all these forms would increase adoption of commercial apps in clinical practice, inform the development of the next generation of apps, and ultimately increase the impact of commercial apps.

[1]  Ronald Tamler,et al.  An evaluation of diabetes self-management applications for Android smartphones , 2012, Journal of telemedicine and telecare.

[2]  Urs-Vito Albrecht,et al.  Transparency of Health-Apps for Trust and Decision Making , 2013, Journal of medical Internet research.

[3]  Amy P Abernethy,et al.  Rapid, responsive, relevant (R3) research: a call for a rapid learning health research enterprise , 2013, Clinical and Translational Medicine.

[4]  Julie A Kientz,et al.  Randomized, controlled pilot trial of a smartphone app for smoking cessation using acceptance and commitment therapy. , 2014, Drug and alcohol dependence.

[5]  E. Rimm,et al.  Gotta catch’em all! Pokémon GO and physical activity among young adults: difference in differences study , 2016, British Medical Journal.

[6]  Martin F Mendiola,et al.  Valuable Features in Mobile Health Apps for Patients and Consumers: Content Analysis of Apps and User Ratings , 2015, JMIR mHealth and uHealth.

[7]  Lori Wozney,et al.  A Systematic Review of Cognitive Behavioral Therapy and Behavioral Activation Apps for Depression , 2016, PloS one.

[8]  John Torous,et al.  Navigating Ethics in the Digital Age: Introducing Connected and Open Research Ethics (CORE), a Tool for Researchers and Institutional Review Boards , 2017, Journal of medical Internet research.

[9]  R. Dellavalle,et al.  Mobile medical and health apps: state of the art, concerns, regulatory control and certification , 2014, Online journal of public health informatics.

[10]  Charmian Reynoldson,et al.  Assessing the quality and usability of smartphone apps for pain self-management. , 2014, Pain medicine.

[11]  R. Istepanian,et al.  M-Health: Emerging Mobile Health Systems , 2006 .

[12]  Anu Kankainen,et al.  Usability testing of mobile applications: a comparison between laboratory and field testing , 2005 .

[13]  Robin Henderson,et al.  Dynamic modelling of n-of-1 data: powerful and flexible data analytics applied to individualised studies , 2017, Health psychology review.

[14]  Devin Mann,et al.  Evidence-based strategies in weight-loss mobile apps. , 2013, American journal of preventive medicine.

[15]  P. Krebs,et al.  Health App Use Among US Mobile Phone Owners: A National Survey , 2015, JMIR mHealth and uHealth.

[16]  D. Johnston,et al.  The state of the art and future opportunities for using longitudinal n-of-1 methods in health behaviour research: a systematic literature overview , 2017, Health psychology review.

[17]  J. Brug,et al.  Apps to promote physical activity among adults: a review and content analysis , 2014, International Journal of Behavioral Nutrition and Physical Activity.

[18]  Alain Abran,et al.  Usability evaluation of mobile applications using ISO 9241 and ISO 25062 standards , 2016, SpringerPlus.

[19]  N. Schork,et al.  The n-of-1 clinical trial: the ultimate strategy for individualizing medicine? , 2011, Personalized medicine.

[20]  M. Johnston,et al.  From Theory to Intervention: Mapping Theoretically Derived Behavioural Determinants to Behaviour Change Techniques , 2008 .

[21]  Hsiu-Fang Hsieh,et al.  Three Approaches to Qualitative Content Analysis , 2005, Qualitative health research.

[22]  Derek Flood,et al.  Usability of mobile applications: literature review and rationale for a new usability model , 2013, Journal of Interaction Science.

[23]  D. Conroy,et al.  Behavior change techniques in top-ranked mobile apps for physical activity. , 2014, American journal of preventive medicine.

[24]  C. Abraham,et al.  A taxonomy of behavior change techniques used in interventions. , 2008, Health psychology : official journal of the Division of Health Psychology, American Psychological Association.

[25]  John Torous,et al.  The digital placebo effect: mobile mental health meets clinical psychiatry. , 2016, The lancet. Psychiatry.

[26]  Philippe Kruchten,et al.  Real Challenges in Mobile App Development , 2013, 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement.

[27]  Christopher M. Wharton,et al.  Dietary self-monitoring, but not dietary quality, improves with use of smartphone app technology in an 8-week weight loss trial. , 2014, Journal of nutrition education and behavior.

[28]  S. Barlow,et al.  Expert committee recommendations regarding the prevention, assessment, and treatment of child and adolescent overweight and obesity: summary report. , 2007, Pediatrics.

[29]  Urs-Vito Albrecht,et al.  Evaluation of mHealth Applications Quality Based on User Ratings , 2016, ICIMTH.

[30]  M T Baysari,et al.  Mobile Applications for Patient-centered Care Coordination: A Review of Human Factors Methods Applied to their Design, Development, and Evaluation , 2015, Yearbook of Medical Informatics.

[31]  Inbal Nahum-Shani,et al.  Optimization of behavioral dynamic treatment regimens based on the sequential, multiple assignment, randomized trial (SMART) , 2014, Clinical trials.

[32]  S. Michie,et al.  Using the Internet to Promote Health Behavior Change: A Systematic Review and Meta-analysis of the Impact of Theoretical Basis, Use of Behavior Change Techniques, and Mode of Delivery on Efficacy , 2010, Journal of medical Internet research.

[33]  Rhonda K. Reger,et al.  A Content Analysis of the Content Analysis Literature in Organization Studies: Research Themes, Data Sources, and Methodological Refinements , 2007 .

[34]  The Diabetes Prevention Program (DPP): description of lifestyle intervention. , 2002, Diabetes care.

[35]  Sara Wilcox,et al.  Mobile apps for pediatric obesity prevention and treatment, healthy eating, and physical activity promotion: just fun and games? , 2013, Translational behavioral medicine.

[36]  M. Mehl,et al.  Handbook of research methods for studying daily life , 2012 .

[37]  Urs-Vito Albrecht,et al.  Evaluation of mHealth Applications Security Based on Application Permissions , 2016, ICIMTH.

[38]  Patrick Crutchley,et al.  Seeing the “Big” Picture: Big Data Methods for Exploring Relationships Between Usage, Language, and Outcome in Internet Intervention Data , 2016, Journal of medical Internet research.

[39]  Melinda D. Maryniuk,et al.  AADE7 Self-Care Behaviors. , 2008, The Diabetes educator.

[40]  Gabrielle M. Turner-McGrievy,et al.  Choosing between responsive-design websites versus mobile apps for your mobile behavioral intervention: presenting four case studies , 2017, Translational behavioral medicine.

[41]  A. Stone,et al.  Strategies for analyzing ecological momentary assessment data. , 1998, Health psychology : official journal of the Division of Health Psychology, American Psychological Association.

[42]  Ali Idri,et al.  Empirical Studies on Usability of mHealth Apps: A Systematic Literature Review , 2015, Journal of Medical Systems.

[43]  Victor Nassar,et al.  Common criteria for usability review. , 2012, Work.

[44]  Chi-Hong Tseng,et al.  Effectiveness of a Smartphone Application for Weight Loss Compared With Usual Care in Overweight Primary Care Patients , 2014, Annals of Internal Medicine.

[45]  S. Michie,et al.  Does theory influence the effectiveness of health behavior interventions? Meta-analysis. , 2014, Health psychology : official journal of the Division of Health Psychology, American Psychological Association.

[46]  Isabel de la Torre Díez,et al.  Experiences and Results of Applying Tools for Assessing the Quality of a mHealth App Named Heartkeeper , 2015, Journal of Medical Systems.

[47]  Kyle Montague,et al.  TinyBlackBox: Supporting Mobile In-The-Wild Studies , 2015, ASSETS.

[48]  David W. Bates,et al.  Usability of Commercially Available Mobile Applications for Diverse Patients , 2016, Journal of General Internal Medicine.

[49]  Li Liang,et al.  An Initial Evaluation of the Impact of Pokémon GO on Physical Activity , 2017, Journal of the American Heart Association.

[50]  Dongsong Zhang,et al.  Challenges, Methodologies, and Issues in the Usability Testing of Mobile Applications , 2005, Int. J. Hum. Comput. Interact..

[51]  S. Michie,et al.  A refined taxonomy of behaviour change techniques to help people change their physical activity and healthy eating behaviours: The CALO-RE taxonomy , 2011, Psychology & health.

[52]  Molly E Waring,et al.  Evaluating and selecting mobile health apps: strategies for healthcare providers and healthcare organizations , 2014, Translational behavioral medicine.

[53]  Michael Miller,et al.  Dietary Fats and Cardiovascular Disease: A Presidential Advisory From the American Heart Association , 2017, Circulation.

[54]  Lyndal Trevena,et al.  A systematic review of quality assessment methods for smartphone health apps. , 2015, Telemedicine journal and e-health : the official journal of the American Telemedicine Association.

[55]  John Torous,et al.  Towards a Framework for Evaluating Mobile Mental Health Apps. , 2015, Telemedicine journal and e-health : the official journal of the American Telemedicine Association.

[56]  S. Murphy,et al.  The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): new methods for more potent eHealth interventions. , 2007, American journal of preventive medicine.

[57]  D. Sackett,et al.  Evidence based medicine: what it is and what it isn't , 1996, BMJ.

[58]  E. Chiauzzi,et al.  ‘Trust but verify’ – five approaches to ensure safe medical apps , 2015, BMC Medicine.

[59]  A. Powell,et al.  Interrater Reliability of mHealth App Rating Measures: Analysis of Top Depression and Smoking Cessation Apps , 2016, JMIR mHealth and uHealth.

[60]  Urs-Vito Albrecht,et al.  Medical Apps -The Road To Trust , 2015 .

[61]  Christopher H Schmid,et al.  Individual (N-of-1) trials can be combined to give population comparative treatment effect estimates: methodologic considerations. , 2010, Journal of clinical epidemiology.

[62]  Jessica Y. Breland,et al.  Adherence to evidence-based guidelines among diabetes self-management apps , 2013, Translational behavioral medicine.

[63]  Audie A Atienza,et al.  Mobile health technology evaluation: the mHealth evidence workshop. , 2013, American journal of preventive medicine.

[64]  Jan Stage,et al.  It's worth the hassle!: the added value of evaluating the usability of mobile systems in the field , 2006, NordiCHI '06.

[65]  Ryen W. White,et al.  Influence of Pokémon Go on Physical Activity: Study and Implications , 2016, Journal of medical Internet research.

[66]  Bettina B. Hoeppner,et al.  How Smart are Smartphone Apps for Smoking Cessation? A Content Analysis. , 2016, Nicotine & tobacco research : official journal of the Society for Research on Nicotine and Tobacco.

[67]  C. Abraham,et al.  The Behavior Change Technique Taxonomy (v1) of 93 Hierarchically Clustered Techniques: Building an International Consensus for the Reporting of Behavior Change Interventions , 2013, Annals of behavioral medicine : a publication of the Society of Behavioral Medicine.

[68]  Daniel E. Rivera,et al.  Agile science: creating useful products for behavior change in the real world , 2016, Translational behavioral medicine.

[69]  Mandi Yu,et al.  Mining Health App Data to Find More and Less Successful Weight Loss Subgroups , 2016, Journal of medical Internet research.