Developing and Applying a Formative Evaluation Framework for Health Information Technology Implementations: Qualitative Investigation

Background There is currently a lack of comprehensive, intuitive, and usable formative evaluation frameworks for health information technology (HIT) implementations. We therefore sought to develop and apply such a framework. This study describes the Technology, People, Organizations, and Macroenvironmental factors (TPOM) framework we developed. Objective The aim was to develop and apply a formative evaluation framework for HIT implementations, highlighting interrelationships between identified dimensions and offering guidance for implementers. Methods We drew on an initial prototype framework developed as part of a literature review exploring factors for the effective implementation of HIT. In addition, we used qualitative data from three national formative evaluations of different HIT interventions (electronic health record, electronic prescribing, and clinical decision support functionality). The combined data set comprised 19 case studies of primarily hospital settings, and included 703 semistructured interviews, 663 hours of observations, and 864 documents gathered from a range of care settings across National Health Service (NHS) England and NHS Scotland. Data analysis took place over a period of 10 years and was guided by a framework informed by the existing evidence base. Results TPOM dimensions are intimately related and each include a number of subthemes that evaluators need to consider. Although technological functionalities are crucial in getting an initiative off the ground, system design needs to be cognizant of the accompanying social and organizational transformations required to ensure that technologies deliver the desired value for a variety of stakeholders. Wider structural changes, characterized by shifting policy landscapes and markets, influence technologies and the ways they are used by organizations and staff. Conclusions The TPOM framework supports formative evaluations of HIT implementation and digitally enabled transformation efforts. There is now a need for prospective application of the TPOM framework to determine its value.

[1]  D. Bates,et al.  Ten key considerations for the successful implementation and adoption of large-scale health information technology. , 2013, Journal of the American Medical Informatics Association : JAMIA.

[2]  D. Edge,et al.  The social shaping of technology , 1988 .

[3]  LeMai Nguyen,et al.  Electronic health records implementation: An evaluation of information system impact and contingency factors , 2014, Int. J. Medical Informatics.

[4]  Carl May,et al.  Health technology assessment in its local contexts: studies of telehealthcare. , 2003, Social science & medicine.

[5]  D. Bates,et al.  Evaluation of medium-term consequences of implementing commercial computerized physician order entry and clinical decision support prescribing systems in two 'early adopter' hospitals. , 2014, Journal of the American Medical Informatics Association : JAMIA.

[6]  Peter B. Seddon,et al.  A Comprehensive Framework for Classifying the Benefits of ERP Systems , 2000 .

[7]  E. Rogers,et al.  Diffusion of innovations , 1964, Encyclopedia of Sport Management.

[8]  A. Sheikh,et al.  Evaluating eHealth Interventions: The Need for Continuous Systemic Evaluation , 2009, PLoS medicine.

[9]  Aziz Sheikh,et al.  The NHS Care Record Service (NHS CRS): recommendations from the literature on successful implementation and adoption. , 2009, Informatics in primary care.

[10]  Sarah Muttitt,et al.  A proposed benefits evaluation framework for health information systems in Canada. , 2007, Healthcare quarterly.

[11]  G. Ritzer.  Structuration Theory , 2007 .

[12]  L. Carroll,et al.  The complete illustrated works , 1995 .

[13]  Eric C. Pan,et al.  Organizational framework for health information technology , 2013, Int. J. Medical Informatics.

[14]  Michelle L. Rogers,et al.  Health Information Technology Evaluation Framework (HITREF) Comprehensiveness as Assessed in Electronic Point-of-Care Documentation Systems Evaluations , 2015, MedInfo.

[15]  Suzanne Bakken,et al.  Review of health information technology usability study methodologies , 2011, J. Am. Medical Informatics Assoc..

[16]  D. Blumenthal.  Launching HITECH. , 2010, The New England journal of medicine.

[17]  Trisha Greenhalgh,et al.  Beyond Adoption: A New Framework for Theorizing and Evaluating Nonadoption, Abandonment, and Challenges to the Scale-Up, Spread, and Sustainability of Health and Care Technologies , 2017, Journal of medical Internet research.

[18]  Tony Cornford,et al.  Experience with a structure, process and outcome framework for evaluating an information system , 1994 .

[19]  Dean F Sittig,et al.  A new sociotechnical model for studying health information technology in complex adaptive healthcare systems , 2010, Quality and Safety in Health Care.

[20]  David Wainwright,et al.  Benefits Realisation of Information Technology in the National Health Service: A Paradigmatic Review , 2015 .

[21]  A. Sheikh,et al.  NHS Scotland’s Decision Support Platform: a formative qualitative evaluation , 2019, BMJ Health & Care Informatics.

[22]  C. Anandan,et al.  The Impact of eHealth on the Quality and Safety of Health Care: A Systematic Overview , 2011, PLoS medicine.

[23]  A. Sheikh,et al.  Undertaking sociotechnical evaluations of health information technologies. , 2014, Informatics in primary care.

[24]  Kathrin Cresswell.  Evaluation of Implementation of Health IT. , 2016, Studies in health technology and informatics.

[25]  D. Bates,et al.  Exploring the roots of unintended safety threats associated with the introduction of hospital ePrescribing systems and candidate avoidance and/or mitigation strategies: a qualitative study , 2017, BMJ Quality & Safety.

[26]  Wendy L. Currie,et al.  It-Enabled Healthcare Delivery: The U.K. National Health Service , 2006, Inf. Syst. Manag..

[27]  David W. Bates,et al.  The use of health information technology in seven nations , 2008, Int. J. Medical Informatics.

[28]  J. Fereday,et al.  Demonstrating Rigor Using Thematic Analysis: A Hybrid Approach of Inductive and Deductive Coding and Theme Development , 2006 .

[29]  M. Marshall.  Sampling for qualitative research. , 1996, Family practice.

[30]  Dimitra I. Petrakaki,et al.  Implementation and adoption of nationwide electronic health records in secondary care in England: final qualitative results from a prospective national evaluation in 'early adopter' hospitals , 2011 .

[31]  A. Sheikh,et al.  Workarounds to hospital electronic prescribing systems: a qualitative study in English hospitals , 2016, BMJ Quality & Safety.

[32]  Aziz Sheikh,et al.  Organizational issues in the implementation and adoption of health information technology innovations: An interpretative review , 2013, Int. J. Medical Informatics.

[33]  W. Baine,et al.  The Agency for Healthcare Research and Quality , 2006, Italian Journal of Public Health.

[34]  P. Sandercock,et al.  Framework for design and evaluation of complex interventions to improve health , 2000, BMJ : British Medical Journal.

[35]  K. A. McKibbon,et al.  Sampling in Qualitative Research: Insights from an Overview of the Methods Literature , 2015 .

[36]  Helana Scheepers,et al.  Health information systems evaluation frameworks: A systematic review , 2017, Int. J. Medical Informatics.

[37]  Anastasia Papazafeiropoulou,et al.  An evaluation framework for Health Information Systems: human, organization and technology-fit factors (HOT-fit) , 2008, Int. J. Medical Informatics.

[38]  Suzanne Bakken,et al.  Applying an Evaluation Framework for Health Information System Design, Development, and Implementation , 2006, Nursing research.