Edinburgh Research Explorer Online Guide for Electronic Health Evaluation Approaches: Systematic Scoping Review and Concept Mapping Study

Background: Despite the increase in use and high expectations of digital health solutions, scientific evidence about the effectiveness of electronic health (eHealth) and other aspects such as usability and accuracy is lagging behind. eHealth solutions are complex interventions, which require a wide array of evaluation approaches that are capable of answering the many different questions that arise during the consecutive study phases of eHealth development and implementation. However, evaluators seem to struggle in choosing Conclusions: Seventy-five unique evaluation approaches were found in the literature and suggested by eHealth researchers, which served as content for the online “eHealth methodology guide.” By assisting evaluators in selecting a suitable evaluation approach in relation to a specific study phase of the “eHealth evaluation cycle,” the guide aims to enhance the quality, safety, and successful long-term implementation of novel eHealth solutions.

[1]  Laurence Alpay,et al.  Embedding persuasive design for self-health management systems in Dutch healthcare informatics education: Application of a theory-based method , 2018, Health Informatics J..

[2]  Jeremy C. Wyatt,et al.  How can clinicians, specialty societies and others evaluate and improve the quality of apps for patient use? , 2018, BMC Medicine.

[3]  Henrik Eriksson,et al.  Evidence-Based Evaluation of eHealth Interventions: Systematic Literature Review , 2018, Journal of medical Internet research.

[4]  Kumanan Wilson,et al.  Agile research to complement agile development: a proposal for an mHealth research lifecycle , 2018, npj Digital Medicine.

[5]  Clayon B Hamilton,et al.  Accuracy of Fitbit Devices: Systematic Review and Narrative Syntheses of Quantitative Data , 2018, JMIR mHealth and uHealth.

[6]  T. Timpka,et al.  mHealth Self-Report Monitoring in Competitive Middle- and Long-Distance Runners: Qualitative Study of Long-Term Use Intentions Using the Technology Acceptance Model , 2018, JMIR mHealth and uHealth.

[7]  S. Murphy,et al.  Optimizing Digital Integrated Care via Micro‐Randomized Trials , 2018, Clinical pharmacology and therapeutics.

[8]  P. Stolee,et al.  Economic evaluations of eHealth technologies: A systematic review , 2018, PloS one.

[9]  Mohammad Mehdi Sepehri,et al.  A framework for m-health service development and success evaluation , 2018, Int. J. Medical Informatics.

[10]  J. A. Oliveira,et al.  The Impact of mHealth Interventions: Systematic Review of Systematic Reviews , 2018, JMIR mHealth and uHealth.

[11]  Kristian Kidholm,et al.  Validity of the Model for Assessment of Telemedicine: A Delphi study , 2016, Journal of telemedicine and telecare.

[12]  I. Bongiovanni-Delarozière,et al.  Economic evaluation methods applied to telemedicine: From a literature review to a standardized framework , 2017 .

[13]  Trisha Greenhalgh,et al.  Beyond Adoption: A New Framework for Theorizing and Evaluating Nonadoption, Abandonment, and Challenges to the Scale-Up, Spread, and Sustainability of Health and Care Technologies , 2017, Journal of medical Internet research.

[14]  Victoria Wade,et al.  Mixed methods for telehealth research , 2017, Journal of telemedicine and telecare.

[15]  A. Maeder,et al.  How to formulate research questions and design studies for telehealth assessment and evaluation , 2017, Journal of telemedicine and telecare.

[16]  Floor Sieverink,et al.  Opening the Black Box of Electronic Health: Collecting, Analyzing, and Interpreting Log Data , 2017, JMIR research protocols.

[17]  Jane Clemensen,et al.  The Model for Assessment of Telemedicine (MAST): A scoping review of empirical studies , 2017, Journal of telemedicine and telecare.

[18]  Liam J Caffery,et al.  Using survey methods in telehealth research: A practical guide , 2017, Journal of telemedicine and telecare.

[19]  Reza Khajouei,et al.  Comparison of heuristic and cognitive walkthrough usability evaluation methods for evaluating health information systems , 2017, J. Am. Medical Informatics Assoc..

[20]  William M. K. Trochim,et al.  Introduction to a special issue on concept mapping. , 2017, Evaluation and program planning.

[21]  Jane Clemensen,et al.  Participatory design methods in telemedicine research , 2016, Journal of telemedicine and telecare.

[22]  Anthony C. Smith,et al.  A systematic review of the methodologies used to evaluate telemedicine service initiatives in hospital facilities , 2017, Int. J. Medical Informatics.

[23]  Monique W. M. Jaspers,et al.  What is needed to implement a web-based audit and feedback intervention with outreach visits to improve care quality: A concept mapping study among cardiac rehabilitation teams , 2017, Int. J. Medical Informatics.

[24]  A W Kushniruk,et al.  Methods for Addressing Technology-induced Errors: The Current State , 2016, Yearbook of Medical Informatics.

[25]  Elizabeth Murray,et al.  Evaluating Digital Health Interventions: Key Questions and Approaches. , 2016, American journal of preventive medicine.

[26]  Amanda L. Graham,et al.  Iterative development and evaluation methods of mHealth behavior change interventions , 2016 .

[27]  Johanna Kaipio,et al.  Quality of Health IT Evaluations. , 2016, Studies in health technology and informatics.

[28]  H. Christensen,et al.  mHealth in psychiatry: time for methodological change , 2016, Evidence-Based Mental Health.

[29]  Lisa M. Law,et al.  Use of an embedded, micro-randomised trial to investigate non-compliance in telehealth interventions , 2016, Clinical trials.

[30]  D. Spiegelman Evaluating Public Health Interventions: 2. Stepping Up to Routine Public Health Evaluation With the Stepped Wedge Design. , 2016, American journal of public health.

[31]  Gregory Campbell,et al.  Statistical innovations in the medical device world sparked by the FDA , 2016, Journal of biopharmaceutical statistics.

[32]  Inbal Nahum-Shani,et al.  Randomised trials for the Fitbit generation , 2015, Significance.

[33]  Ambuj Tewari,et al.  Microrandomized trials: An experimental design for developing just-in-time adaptive interventions. , 2015, Health psychology : official journal of the Division of Health Psychology, American Psychological Association.

[34]  R. Lawton,et al.  What methods are used to apply positive deviance within healthcare organisations? A systematic review , 2015, BMJ Quality & Safety.

[35]  A. Farmer,et al.  Interactive telemedicine: effects on professional practice and health care outcomes. , 2015, The Cochrane database of systematic reviews.

[36]  William T Riley,et al.  Trials of Intervention Principles: Evaluation Methods for Evolving Behavioral Intervention Technologies , 2015, Journal of medical Internet research.

[37]  S. de Lusignan,et al.  Creating and using real-world evidence to answer questions about clinical effectiveness , 2015, BMJ Health & Care Informatics.

[38]  Evangelos Kontopantelis,et al.  Regression based quasi-experimental approach when randomisation is not an option: interrupted time series analysis , 2015, BMJ : British Medical Journal.

[39]  Miranda Olff,et al.  Mobile mental health: a challenging research agenda , 2015, European journal of psychotraumatology.

[40]  Christian Nøhr,et al.  Clinical simulation: A method for development and evaluation of clinical information systems , 2015, J. Biomed. Informatics.

[41]  Miguel López-Coronado,et al.  Cost-utility and cost-effectiveness studies of telemedicine, electronic, and mobile health systems in the literature: a systematic review. , 2015, Telemedicine journal and e-health : the official journal of the American Telemedicine Association.

[42]  Nicolette de Keizer,et al.  How to use concept mapping to identify barriers and facilitators of an electronic quality improvement intervention , 2015, MIE.

[43]  Lisa M. Law,et al.  Design of telehealth trials - Introducing adaptive approaches , 2014, Int. J. Medical Informatics.

[44]  Russell E. Glasgow,et al.  Implementation science approaches for integrating eHealth research into practice and policy , 2014, Int. J. Medical Informatics.

[45]  Susan A. Murphy,et al.  Introduction to SMART designs for the development of adaptive interventions: with application to weight loss research , 2014, Translational behavioral medicine.

[46]  H. Raat,et al.  Effectiveness and Cost-Effectiveness of eHealth Interventions in Somatic Diseases: A Systematic Review of Systematic Reviews and Meta-Analyses , 2014, Journal of medical Internet research.

[47]  A. Sheikh,et al.  Undertaking sociotechnical evaluations of health information technologies. , 2014, Informatics in primary care.

[48]  Dhavan V. Shah,et al.  How Can Research Keep Up With eHealth? Ten Strategies for Increasing the Timeliness and Usefulness of eHealth Research , 2014, Journal of medical Internet research.

[49]  David C Mohr,et al.  Continuous evaluation of evolving behavioral intervention technologies. , 2013, American journal of preventive medicine.

[50]  Maurice Mars,et al.  Principles and Framework for eHealth Strategy Development , 2013, Journal of medical Internet research.

[51]  Amy P Abernethy,et al.  Rapid, responsive, relevant (R3) research: a call for a rapid learning health research enterprise , 2013, Clinical and Translational Medicine.

[52]  Jesse Dallery,et al.  Single-Case Experimental Designs to Evaluate Novel Technology-Based Health Interventions , 2013, Journal of medical Internet research.

[53]  Jos Kleijnen,et al.  What is a rapid review? A methodological exploration of rapid reviews in Health Technology Assessments. , 2012, International journal of evidence-based healthcare.

[54]  R. Brownson,et al.  Concept mapping: priority community strategies to create changes to support active living. , 2012, American journal of preventive medicine.

[55]  P. Kummervold,et al.  Using Noninferiority Tests to Evaluate Telemedicine and E-Health Services: Systematic Review , 2012, Journal of medical Internet research.

[56]  Elske Ammenwerth,et al.  Simulation Studies for the Evaluation of Health Information Technologies: Experiences and Results , 2012, Health information management : journal of the Health Information Management Association of Australia.

[57]  Seonah Lee,et al.  Criteria for Quantitative and Qualitative Data Integration: Mixed-Methods Research Methodology , 2012, Computers, informatics, nursing : CIN.

[58]  Alison Bowes,et al.  Methodologies for assessing telemedicine: A systematic review of reviews , 2012, Int. J. Medical Informatics.

[59]  Nicol Nijland,et al.  A Holistic Framework to Improve the Uptake and Impact of eHealth Technologies , 2011, Journal of medical Internet research.

[60]  Kelli L. Kramer-Jackman,et al.  Method for Technology-Delivered Healthcare Measures , 2011, Computers, informatics, nursing : CIN.

[61]  Nicolette de Keizer,et al.  Guideline for good evaluation practice in health informatics (GEP-HI) , 2011, Int. J. Medical Informatics.

[62]  Mona Choi,et al.  Handbook of Evaluation Methods for Health Informatics , 2011, Healthcare Informatics Research.

[63]  Dimitra I. Petrakaki,et al.  Implementation and adoption of nationwide electronic health records in secondary care in England: final qualitative results from a prospective national evaluation in 'early adopter' hospitals , 2011 .

[64]  Jeremy C. Wyatt,et al.  The case for randomized controlled trials to assess the impact of clinical information systems , 2011, J. Am. Medical Informatics Assoc..

[65]  Jeong Yeob Han Transaction logfile analysis in health communication research: challenges and opportunities. , 2011, Patient education and counseling.

[66]  C. Anandan,et al.  The Impact of eHealth on the Quality and Safety of Health Care: A Systematic Overview , 2011, PLoS medicine.

[67]  J. M. Christian Bastien,et al.  Usability testing: a review of some methodological and technical aspects of the method , 2010, Int. J. Medical Informatics.

[68]  Elske Ammenwerth,et al.  Evidence based health informatics. , 2010, Studies in health technology and informatics.

[69]  J. Seeley,et al.  Methodological Issues in Research on Web-Based Behavioral Interventions , 2009, Annals of behavioral medicine : a publication of the Society of Behavioral Medicine.

[70]  A. Sheikh,et al.  Evaluating eHealth Interventions: The Need for Continuous Systemic Evaluation , 2009, PLoS medicine.

[71]  Monique W. M. Jaspers,et al.  A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence , 2009, Int. J. Medical Informatics.

[72]  Recommendations for research design of telehealth studies. , 2008, Telemedicine journal and e-health : the official journal of the American Telemedicine Association.

[73]  M. Petticrew,et al.  Developing and evaluating complex interventions: the new Medical Research Council guidance , 2008, BMJ : British Medical Journal.

[74]  Jan P Vandenbroucke,et al.  Observational Research, Randomised Trials, and Two Views of Medical Science , 2008, PLoS medicine.

[75]  Bonnie Kaplan,et al.  Expanding multi-disciplinary approaches to healthcare information technologies: What does information systems offer medical informatics? , 2007, Int. J. Medical Informatics.

[76]  Russell E Glasgow,et al.  eHealth evaluation and dissemination research. , 2007, American journal of preventive medicine.

[77]  S. Murphy,et al.  The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): new methods for more potent eHealth interventions. , 2007, American journal of preventive medicine.

[78]  Huong Q. Nguyen,et al.  Methodological considerations in evaluating eHealth interventions. , 2007, The Canadian journal of nursing research = Revue canadienne de recherche en sciences infirmieres.

[79]  J. Hughes,et al.  Design and analysis of stepped wedge cluster randomized trials. , 2007, Contemporary clinical trials.

[80]  R. Lilford,et al.  Bmc Medical Research Methodology Open Access the Stepped Wedge Trial Design: a Systematic Review , 2022 .

[81]  Mary Kane,et al.  Concept Mapping for Planning and Evaluation , 2006 .

[82]  J. Grigsby,et al.  Alternatives to randomized controlled trials in telemedicine , 2006, Journal of telemedicine and telecare.

[83]  C. May A rational model for assessing and evaluating complex interventions in health care , 2006, BMC Health Services Research.

[84]  Suzanne Bakken,et al.  Applying an Evaluation Framework for Health Information System Design, Development, and Implementation , 2006, Nursing research.

[85]  J Brender Evaluation of health information applications--challenges ahead of us. , 2006, Methods of information in medicine.

[86]  Vimla L. Patel,et al.  Cognitive and usability engineering methods for the evaluation of clinical information systems , 2004, J. Biomed. Informatics.

[87]  G. Hastings,et al.  The challenge of evaluating complex interventions: a framework for evaluating media advocacy. , 2002, Health education research.

[88]  Nicola T. Shaw,et al.  'CHEATS': a generic information communication technology (ICT) evaluation framework , 2002, Comput. Biol. Medicine.

[89]  Bonnie Kaplan,et al.  Evaluating informatics applications - some alternative approaches: theory, social interactionism, and call for methodological pluralism , 2001, Int. J. Medical Informatics.

[90]  William M. K. Trochim,et al.  An introduction to concept mapping for planning and evaluation. , 1989 .

[91]  William M. K. Trochim,et al.  Conceptualization for Planning and Evaluation , 2022 .