Heuristic Evaluation of Ehealth Interventions: Establishing Standards That Relate to the Therapeutic Process Perspective

In recent years, the number of available eHealth interventions aimed at treating behavioral and mental health challenges has been growing. From the perspective of health care providers, there is a need for eHealth interventions to be evaluated prior to clinical trials and for the limited resources allocated to empirical research to be invested in the most promising products. Following a literature review, a gap was found in the availability of eHealth interventions evaluation principles related to the patient experience of the therapeutic process. This paper introduces principles and concepts for the evaluation of eHealth interventions developed as a first step in a process to outline general evaluation guidelines that relate to the clinical context from health care providers’ perspective. Our approach was to conduct a review of literature that relates to the examination of eHealth interventions. We identified the literature that was most relevant to our study and used it to define guidelines that relate to the clinical context. We then compiled a list of heuristics we found to be useful for the evaluation of eHealth intervention products’ suitability for empirical examination. Four heuristics were identified with respect to the therapeutic process: (1) the product’s ease of use (ie, usability), (2) the eHealth intervention’s compatibility with the clinical setting, (3) the presence of tools that make it easier for the user to engage in therapeutic activities, and (4) the provision of a feasible therapeutic pathway to growth. We then used this set of heuristics to conduct a detailed examination of MyFitnessPal. This line of work could help to set the bar higher for product developers and to inform health care providers about preferred eHealth intervention designs.

[1]  Peter J Schulz,et al.  Medical Information on the Internet: A Tool for Measuring Consumer Perception of Quality Aspects , 2015, Interactive journal of medical research.

[2]  M. Harjumaa,et al.  A Systematic Framework for Designing and Evaluating Persuasive Systems , 2008, PERSUASIVE.

[3]  Maria C Conesa-Fuentes,et al.  Evaluation of the quality of the general health information webpages in Spain: influence of page source , 2013, Informatics for health & social care.

[4]  C. Flavián,et al.  Perceived e-service quality ( PeSQ ) Measurement validation and effects on consumer satisfaction and web site loyalty , 2012 .

[5]  Boris P. Kovatchev,et al.  A Behavior Change Model for Internet Interventions , 2009, Annals of behavioral medicine : a publication of the Society of Behavioral Medicine.

[6]  Mike Thelwall,et al.  Online Interventions for Social Marketing Health Behavior Change Campaigns: A Meta-Analysis of Psychological Architectures and Adherence Factors , 2011, Journal of medical Internet research.

[7]  David C Mohr,et al.  Continuous evaluation of evolving behavioral intervention technologies. , 2013, American journal of preventive medicine.

[8]  Felix Naumann,et al.  Assessment Methods for Information Quality Criteria , 2000, IQ.

[9]  Avi Parush,et al.  Building it better: Applying human-computer interaction and persuasive system design principles to a monetary limit tool improves responsible gambling , 2014, Comput. Hum. Behav..

[10]  Brian McMillan,et al.  Quality assessment of a sample of mobile app-based health behavior change interventions using a tool based on the National Institute of Health and Care Excellence behavior change guidance. , 2016, Patient education and counseling.

[11]  Parisa Rashidi,et al.  The Behavioral Intervention Technology Model: An Integrated Conceptual and Technological Framework for eHealth and mHealth Interventions , 2014, Journal of medical Internet research.

[12]  Angela M. White,et al.  Assessment of function and clinical utility of alcohol and other drug web sites: An observational, qualitative study , 2011, BMC public health.

[13]  R. Hatcher,et al.  Patients' view of the alliance of psychotherapy: exploratory factor analysis of three alliance measures. , 1996, Journal of consulting and clinical psychology.

[14]  K. Patrick,et al.  Evaluation of physical activity web sites for use of behavior change theories , 2003, Annals of behavioral medicine : a publication of the Society of Behavioral Medicine.

[15]  Gustavo Rossi,et al.  Measuring Web Application Quality with WebQEM , 2002, IEEE Multim..

[16]  Gbogboade Ademiluyi,et al.  Evaluating the reliability and validity of three tools to assess the quality of health information on the Internet. , 2003, Patient education and counseling.

[17]  Audie A Atienza,et al.  Mobile health technology evaluation: the mHealth evidence workshop. , 2013, American journal of preventive medicine.

[18]  Jakob Nielsen,et al.  Ten Usability Heuristics , 2006 .

[19]  Meng-Chieh Chiu,et al.  Motivating the motivators: Lessons learned from the design and evaluation of a social persuasion system , 2014, Pervasive Mob. Comput..

[20]  R. Elliott,et al.  Measuring the impact of sessions in process€xperiential therapy of depression: The Session Impacts Scale. , 1994 .

[21]  Predrag V. Klasnja,et al.  Mind the theoretical gap: interpreting, using, and developing behavioral theory in HCI research , 2013, CHI.

[22]  Virpi Roto,et al.  Developing practical tools for user experience evaluation: a case from mobile news journalism , 2009, ECCE.

[23]  S. Michie,et al.  Using the Internet to Promote Health Behavior Change: A Systematic Review and Meta-analysis of the Impact of Theoretical Basis, Use of Behavior Change Techniques, and Mode of Delivery on Efficacy , 2010, Journal of medical Internet research.

[24]  Paul Kim,et al.  Published criteria for evaluating health related web sites: review , 1999, BMJ.

[25]  Begoña García Zapirain,et al.  Security Recommendations for mHealth Apps: Elaboration of a Developer’s Guide , 2016, Journal of Medical Systems.

[26]  Jaeho Lee,et al.  Developing a Framework for Evaluating the Patient Engagement, Quality, and Safety of Mobile Health Applications. , 2016, Issue brief.

[27]  Shifali Arora,et al.  Privacy and Security in Mobile Health (mHealth) Research , 2014, Alcohol research : current reviews.

[28]  Maria C Conesa-Fuentes,et al.  Prospective analysis of the quality of Spanish health information web sites after 3 years , 2016, Informatics for health & social care.

[29]  Pietro Ghezzi,et al.  A Methodology to Analyze the Quality of Health Information on the Internet , 2015, The Diabetes educator.

[30]  Luca Chittaro,et al.  Interactive Persuasive Systems: A Perspective on Theory and Evaluation , 2016, Int. J. Hum. Comput. Interact..

[31]  P. Cuijpers,et al.  Supportive Accountability: A Model for Providing Human Support to Enhance Adherence to eHealth Interventions , 2011, Journal of medical Internet research.

[32]  V P Abbott,et al.  Web page quality: can we measure it and what do we find? A report of exploratory findings. , 2000, Journal of public health medicine.

[33]  Dennis L. Eggett,et al.  Measuring the Relevance of Evaluation Criteria among Health Information Seekers on the Internet , 2003, Journal of health psychology.

[34]  Juliana Chen,et al.  The Most Popular Smartphone Apps for Weight Loss: A Quality Assessment , 2015, JMIR mHealth and uHealth.

[35]  Heidi Krömker,et al.  A framework to measure user experience of interactive online products , 2010, MB '10.

[36]  M. Blais Development of an Inpatient Treatment Alliance Scale , 2004, The Journal of nervous and mental disease.

[37]  Johannes Brug,et al.  Dutch Young Adults Ratings of Behavior Change Techniques Applied in Mobile Phone Apps to Promote Physical Activity: A Cross-Sectional Survey , 2015, JMIR mHealth and uHealth.

[38]  Viswanath Venkatesh,et al.  Assessing a Firm's Web Presence: A Heuristic Evaluation Procedure for the Measurement of Usability , 2002, Inf. Syst. Res..

[39]  S. Kelders,et al.  Persuasive System Design Does Matter: A Systematic Review of Adherence to Web-Based Interventions , 2012, Journal of medical Internet research.

[40]  Susie Skarl,et al.  Anxiety and Depression Association of America , 2015 .

[41]  C. Hoge,et al.  PTSD treatment for soldiers after combat deployment: low utilization of mental health care and reasons for dropout. , 2014, Psychiatric services.

[42]  L. Davidson,et al.  Preliminary Evaluation of the Validity and Reliability of the Spanish Version of the Therapeutic Alliance with Clinician (TAC) Questionnaire , 2006 .

[43]  Alan T. Murray,et al.  Geographic influences on sexual and reproductive health service utilization in rural Mozambique. , 2012, Applied geography.

[44]  T. Corbett,et al.  Behavior Change Techniques in Apps for Medication Adherence: A Content Analysis. , 2016, American journal of preventive medicine.

[45]  O. Adorisio,et al.  Analysis of Readability and Quality of Web Pages Addressing both Common and Uncommon Topics in Pediatric Surgery , 2012, European Journal of Pediatric Surgery.

[46]  Y. Aljeesh,et al.  Predictors of Acute Diarrhoea among Hospitalized Children in Gaza Governorates: A Case-Control Study , 2015, Journal of health, population, and nutrition.

[47]  Shahriar Akter,et al.  Development and validation of an instrument to measure user perceived service quality of mHealth , 2013, Inf. Manag..

[48]  A. Sheikh,et al.  Evaluating eHealth Interventions: The Need for Continuous Systemic Evaluation , 2009, PLoS medicine.

[49]  Alessandro D'Atri,et al.  A quality evaluation methodology of health web-pages for non-professionals , 2004, Medical informatics and the Internet in medicine.

[50]  M. Feki,et al.  Plasma arachidonic and docosahexaenoic acids in Tunisian very low birth weight infants: status and association with selected neonatal morbidities , 2015, Journal of Health, Population and Nutrition.

[51]  Nicolae-George Dragulanescu,et al.  Website Quality Evaluations: Criteria and Tools , 2002 .

[52]  Bradley C. Martin,et al.  The initial development of the WebMedQual scale: Domain assessment of the construct of quality of health web sites , 2006, Int. J. Medical Informatics.

[53]  S. Michie,et al.  Behavior Change Techniques in Popular Alcohol Reduction Apps: Content Analysis , 2015, Journal of medical Internet research.

[54]  A. Horvath,et al.  Development and validation of the Working Alliance Inventory. , 1989 .

[55]  A. Garland,et al.  Psychometric properties of the Therapeutic Alliance Scale for Caregivers and Parents. , 2013, Psychological assessment.

[56]  W. Nilsen,et al.  Health behavior models in the age of mobile interventions: are our theories up to the task? , 2011, Translational behavioral medicine.

[57]  Ana Belén Martínez Prieto,et al.  Sirius: A heuristic-based framework for measuring web usability adapted to the type of website , 2013, J. Syst. Softw..

[58]  M. Gulliford,et al.  Geographical inequalities in lung cancer management and survival in South East England: evidence of variation in access to oncology services? , 2003, British Journal of Cancer.

[59]  Vassilis Moustakis,et al.  Website Quality Assessment Criteria , 2004, ICIQ.

[60]  Elizabeth M. Borycki,et al.  Issues and Considerations for Healthcare Consumers Using Mobile Applications , 2013, ITCH.

[61]  Clare-Marie Karat,et al.  A comparison of user interface evaluation methods , 1994 .

[62]  Nancy Staggers,et al.  A Modified User-Oriented Heuristic Evaluation of a Mobile Health System for Diabetes Self-management Support , 2016, Computers, informatics, nursing : CIN.

[63]  Isabel de la Torre Díez,et al.  Development and Evaluation of Tools for Measuring the Quality of Experience (QoE) in mHealth Applications , 2013, Journal of Medical Systems.

[64]  Nor Aziah Daud,et al.  An Initial Model of Persuasive Design in Web based Learning Environment , 2013 .

[65]  B. J. Fogg,et al.  A behavior model for persuasive design , 2009, Persuasive '09.

[66]  Ashley Fisher,et al.  Review of mobile health technology for military mental health. , 2014, Military medicine.

[67]  Jakob Nielsen,et al.  Enhancing the explanatory power of usability heuristics , 1994, CHI '94.

[68]  M. Koblinsky,et al.  Predictors of maternal health services utilization by poor, rural women: a comparative study in Indian States of Gujarat and Tamil Nadu , 2015, Journal of Health, Population and Nutrition.

[69]  Deborah I Thompson,et al.  Design of Video Games for Children's Diet and Physical Activity Behaviour Change , 2010, Int. J. Comput. Sci. Sport.

[70]  Wayne F. Velicer,et al.  Development of Review Criteria to Evaluate Health Behavior Change Websites , 2003, Journal of health psychology.

[71]  Noam Tractinsky,et al.  Assessing dimensions of perceived visual aesthetics of web sites , 2004 .

[72]  Peter A. Bath,et al.  Assessing the quality of websites providing information on multiple sclerosis: evaluating tools and comparing sites , 2007, Health Informatics J..

[73]  Examination of Therapeutic Alliance and Dependent-Care Agency in the Context of Complementary and Alternative Therapy Utilization by Mothers for Their Children with Asthma , 2002 .

[74]  Ali Idri,et al.  A Framework for Evaluating the Software Product Quality of Pregnancy Monitoring Mobile Personal Health Records , 2016, Journal of Medical Systems.

[75]  Johannes Mander The individual therapy process questionnaire: development and validation of a revised measure to evaluate general change mechanisms in psychotherapy. , 2015, Clinical psychology & psychotherapy.

[76]  Paula Savioja,et al.  European Conference on Cognitive Ergonomics: Designing beyond the Product --- Understanding Activity and User Experience in Ubiquitous Environments , 2009 .

[77]  Sanjeev Sockalingam,et al.  Analysis of the Information Quality of Bariatric Surgery Smartphone Applications Using the Silberg Scale , 2015, Obesity Surgery.

[78]  Scott D. Miller,et al.  The Session Rating Scale: Preliminary Psychometric Properties of a “Working” Alliance Measure , 2004 .

[79]  Barbara M. Wildemuth,et al.  An Assessment of the Quality and Usability of Smoking Cessation Information on the Internet , 2003, Health promotion practice.

[80]  M. Kuehnhausen,et al.  Trusting smartphone Apps? To install or not to install, that is the question , 2013, 2013 IEEE International Multi-Disciplinary Conference on Cognitive Methods in Situation Awareness and Decision Support (CogSIMA).

[81]  W. Stiles,et al.  Alliance structure assessed by the Agnew Relationship Measure (ARM). , 1998, The British journal of clinical psychology.

[82]  Yu-chu Yeh,et al.  Development of design criteria and evaluation scale for web-based learning platforms , 2009 .

[83]  Siow-Li Lai,et al.  Correlates of and Barriers to the Utilization of Health Services for Delivery in South Asia and Sub-Saharan Africa , 2013, TheScientificWorldJournal.

[84]  Kraig Finstad,et al.  The Usability Metric for User Experience , 2010, Interact. Comput..

[85]  Prashant C. Palvia,et al.  Developing and validating an instrument for measuring user-perceived web quality , 2002, Inf. Manag..

[86]  Morten Moshagen,et al.  A short version of the visual aesthetics of websites inventory , 2013 .

[87]  J. Barber,et al.  The Revised Helping Alliance Questionnaire (HAq-II) : Psychometric Properties. , 1996, The Journal of psychotherapy practice and research.

[88]  C. Lançon,et al.  The 4-Point ordinal Alliance Self-report: a self-report questionnaire for assessing therapeutic relationships in routine mental health. , 2009, Comprehensive Psychiatry.

[89]  P. Bower,et al.  A review and critical appraisal of measures of therapist-patient interactions in mental health settings. , 2008, Health technology assessment.

[90]  Heiko Spallek,et al.  English and Spanish oral cancer information on the internet: a pilot surface quality and content evaluation of oral cancer web sites. , 2011, Journal of public health dentistry.

[91]  S. Gozum,et al.  Evaluation of Quality, Content, and Use of the Web Site Prepared for Family Members Giving Care to Stroke Patients , 2015, Computers, informatics, nursing : CIN.

[92]  K. Prasad,et al.  Development and validation of a tool to evaluate the quality of medical education websites in pathology , 2013, Journal of pathology informatics.

[93]  Gordon Parker,et al.  Therapeutic Alliance With a Fully Automated Mobile Phone and Web-Based Intervention: Secondary Analysis of a Randomized Controlled Trial , 2016, JMIR mental health.

[94]  Ravi T. Seethamraju Measurement of user-perceived web quality , 2004, ECIS.

[95]  Sean A. Munson,et al.  Persuasive Performance Feedback: The Effect of Framing on Self-Efficacy , 2013, AMIA.

[96]  Harri Oinas-Kukkonen,et al.  Persuasive Systems Design: Key Issues, Process Model, and System Features , 2009, Commun. Assoc. Inf. Syst..

[97]  Russell A. McCann,et al.  mHealth for mental health: Integrating smartphone technology in behavioral healthcare. , 2011 .

[98]  Suzanne Bakken,et al.  Expanding DISCERN to create a tool for assessing the quality of Web-based health information resources. , 2008, AMIA ... Annual Symposium proceedings. AMIA Symposium.

[99]  Sandra Williams Hilfiker,et al.  Making Quality Health Websites a National Public Health Priority: Toward Quality Standards , 2016, Journal of medical Internet research.

[100]  Eun Kyoung Choe,et al.  Heuristic evaluation of persuasive health technologies , 2010, IHI.

[101]  M. Blais,et al.  Exploring therapeutic alliance in brief inpatient psychotherapy: a preliminary study. , 2010, Clinical psychology & psychotherapy.

[102]  B. J. Fogg Healthy Living with Persuasive Technologies : Framework , Issues , and Challenges , 2009 .

[103]  D. Forman,et al.  Travel time to hospital and treatment for breast, colon, rectum, lung, ovary and prostate cancer. , 2008, European journal of cancer.

[104]  Mable B. Kinzie,et al.  Instructional design strategies for health behavior change. , 2005, Patient education and counseling.

[105]  Oksana Zelenko,et al.  Mobile App Rating Scale: A New Tool for Assessing the Quality of Health Mobile Apps , 2015, JMIR mHealth and uHealth.

[106]  J. Macdermid,et al.  The quality of websites addressing fibromyalgia: an assessment of quality and readability using standardised tools , 2011, BMJ Open.