Design and Evaluation in eHealth: Challenges and Implications for an Interdisciplinary Field

Much has been written about insufficient user involvement in the design of eHealth applications, the lack of evidence demonstrating impact, and the difficulties these bring for adoption. Part of the problem lies in the differing languages, cultures, motives, and operational constraints of producers and evaluators of eHealth systems and services. This paper reflects on the benefits of and barriers to interdisciplinary collaboration in eHealth, focusing particularly on the relationship between software developers and health services researchers. It argues that the common pattern of silo or parallel working may be ameliorated by developing mutual awareness and respect for each others’ methods, epistemologies, and contextual drivers and by recognizing and harnessing potential synergies. Similarities and differences between models and techniques used in both communities are highlighted in order to illustrate the potential for integrated approaches and the strengths of unique paradigms. By sharing information about our research approaches and seeking to actively collaborate in the process of design and evaluation, the aim of achieving technologies that are truly user-informed, fit for context, high quality, and of demonstrated value is more likely to be realized. This may involve embracing new ways of working jointly that are unfamiliar to the stakeholders involved and that challenge disciplinary conventions. It also has policy implications for agencies commissioning research and development in this area.

[1]  H. Rex Hartson,et al.  Toward Empirically Derived Methodologies and Tools for Human-Computer Interface Development , 1989, Int. J. Man Mach. Stud..

[2]  bias in medical informatics evaluation literature : Recognizing the problem , its impact and the causes , 2006 .

[3]  M. Cahn,et al.  Bridging the gap in medical informatics and health services research: workshop results and next steps. , 2002, Journal of the American Medical Informatics Association : JAMIA.

[4]  Bonnie Kaplan,et al.  Evaluating informatics applications - some alternative approaches: theory, social interactionism, and call for methodological pluralism , 2001, Int. J. Medical Informatics.

[5]  Karin A. Thursky,et al.  User-centered design techniques for a computerised antibiotic decision support system in an intensive care unit , 2007, Int. J. Medical Informatics.

[6]  Charles P. Friedman,et al.  White Paper: Toward an Informatics Research Agenda: Key People and Organizational Issues , 2001, J. Am. Medical Informatics Assoc..

[7]  Marion J. Ball,et al.  Evaluation Methods in Biomedical Informatics , 2006 .

[8]  Jeremy C Wyatt,et al.  Evaluation of ehealth systems and services , 2004, BMJ : British Medical Journal.

[9]  Kenneth D. Mandl,et al.  Viewpoint: Integrating Medical Informatics and Health Services Research: The Need for Dual Training at the Clinical Health Systems and Policy Levels , 2002, J. Am. Medical Informatics Assoc..

[10]  Brian Fitzgerald,et al.  An empirical investigation into the adoption of systems development methodologies , 1998, Inf. Manag..

[11]  B. Karsh.  Beyond usability: designing effective technology implementation systems to promote patient safety , 2004, Quality and Safety in Health Care.

[12]  Nicolette de Keizer,et al.  UvA-DARE ( Digital Academic Repository ) An inventory of evaluation studies of information technology in health care trends in evaluation research 1982-2002 , 2005 .

[13]  William Lewis,et al.  Software Testing and Continuous Quality Improvement , 2000 .

[14]  H. McConnell,et al.  International efforts in implementing national health information infrastructure and electronic health records. , 2004, World hospitals and health services : the official journal of the International Hospital Federation.

[15]  Hans-Ulrich Prokosch,et al.  Visions and strategies to improve evaluation of health information systems: Reflections and lessons based on the HIS-EVAL workshop in Innsbruck , 2004, Int. J. Medical Informatics.

[16]  Tom Fahey,et al.  Evaluation of computer based clinical decision support system and risk chart for management of hypertension in primary care: randomised controlled trial , 2000, BMJ : British Medical Journal.

[17]  P. Shekelle,et al.  Systematic Review: Impact of Health Information Technology on Quality, Efficiency, and Costs of Medical Care , 2006, Annals of Internal Medicine.

[18]  P. Robinson,et al.  Competing in the global economy - the innovation challenge , 2004 .

[19]  Marie-Pierre Gagnon,et al.  Striving for evidence in e-health evaluation: Lessons from health technology assessment , 2005, Journal of telemedicine and telecare.

[20]  P. Weingart.  2. Interdisciplinarity: The Paradoxical Discourse , 2000 .

[21]  N. Shaw,et al.  Bad Health Informatics Can Kill – Is Evaluation the Answer? , 2005, Methods of Information in Medicine.

[22]  Claudia Pagliari,et al.  Implementing the National Programme for IT: what can we learn from the Scottish experience? , 2005, Informatics in primary care.

[23]  G Freriks,et al.  Quality management issues for medical ICT. , 2004, Studies in health technology and informatics.

[24]  Annalu Waller,et al.  Participatory design of a text message scheduling system to support young people with diabetes , 2006, Health Informatics J..

[25]  Nancy M Lorenzi.  Beyond the gadgets , 2004, BMJ : British Medical Journal.

[26]  W. Edwards Deming,et al.  Out of the Crisis , 1982 .

[27]  Chris Sauer,et al.  Lessons from a failed information systems initiative: issues for complex organisations , 1999, Int. J. Medical Informatics.

[28]  Charles P. Friedman,et al.  Evaluation Methods in Biomedical Informatics (Health Informatics) , 2005 .

[29]  Barry W. Boehm,et al.  A spiral model of software development and enhancement , 1986, Computer.

[30]  Monique W. M. Jaspers,et al.  The think aloud method: a guide to user interface design , 2004, Int. J. Medical Informatics.

[31]  J. Powell,et al.  Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. , 2002, JAMA.

[32]  P. Sandercock,et al.  Framework for design and evaluation of complex interventions to improve health , 2000, BMJ : British Medical Journal.

[33]  Kweku Ewusi-Mensah,et al.  Software Development Failures , 2003 .

[34]  J. Kahan,et al.  What Is eHealth (4): A Scoping Exercise to Map the Field , 2005, Journal of medical Internet research.

[35]  J Henderson,et al.  Developers and evaluation of interactive health communication applications. The Science Panel on Interactive Communications and Health. , 1999, American journal of preventive medicine.

[36]  R. Hanka,et al.  Evaluating information technology in health care: barriers and challenges , 1998, BMJ.

[37]  M W M Jaspers,et al.  Cognitive engineering in interface design. , 2002, Studies in health technology and informatics.

[38]  Vimla L. Patel,et al.  Usability testing in medical informatics: cognitive approaches to evaluation of information systems and user interfaces , 1997, AMIA.

[39]  E Coiera,et al.  Section 1: Health and Clinical Mangement: The Safety and Quality of Decision Support Systems , 2006, Yearbook of Medical Informatics.

[40]  W. Royce.  Managing the development of large software systems: concepts and techniques , 2021, ICSE '87.

[41]  Charles P. Friedman,et al.  Toward an Informatics Research Agenda , 2001 .

[42]  J. Grigsby,et al.  The evaluation of telemedicine and health services research. , 2005, Telemedicine journal and e-health : the official journal of the American Telemedicine Association.

[43]  Roger Frost,et al.  International Organization for Standardization (ISO) , 2004 .

[44]  Rod R Zalunardo,et al.  Barriers to telemedicine implementation. Usually it's not technology issues that undermine a project--it's everything else. , 2002, Healthcare informatics : the business magazine for information and communication systems.

[45]  Gheorghe Tecuci,et al.  Teaching Intelligent Agents: Software Design Methodology , 2001 .

[46]  U. Reinhardt,et al.  Health care spending and use of information technology in OECD countries. , 2006, Health affairs.