Challenges, solutions and future directions in the evaluation of service innovations in health care and public health

Policies and interventions in the health-care system may have a wide range of effects on multiple patient outcomes and operate through many clinical processes. This presents a challenge for their evaluation, especially when the effect on any one patient is small. In this essay, we explore the nature of the health-care system and discuss how the empirical evidence produced within it relates to the underlying processes governing patient outcomes. We argue for an evidence synthesis framework that first models the underlying phenomena common across different health-care settings and then makes inferences regarding these phenomena from data. Bayesian methods are recommended. We provide the examples of electronic prescribing and increased consultant provision at the weekend.

Jo Rycroft-Malone | Jeremy M. Grimshaw | Jasjeet S. Sekhon | Stephanie L. Prady | Richard Grieve | Rosalind Raine | Graham Moore | Jean-Louis Denis | Nick Freemantle | Richard J. Lilford | Nick Black | Ray Fitzpatrick | Mary Dixon-Woods | Alicia O’Cathain | Marion K Campbell | Peter Bower | Lesley Fallowfield | Merrick Zwarenstein | Naomi Fulop | Ruth Boaden | Chrysanthi Papoutsi | Helen Barratt | Robbie Foy | Ruth McDonald | Laurence Moore | Julien E. Forder | Simon Turner | Zeynep Or | Samuel I Watson | J. Sekhon | N. Freemantle | M. Dixon-Woods | P. Bower | J. Grimshaw | N. Black | R. Grieve | A. O’Cathain | L. Fallowfield | R. Fitzpatrick | K. Devers | G. Moore | M. Zwarenstein | J. Forder | S. Turner | J. Denis | R. Lilford | S. Watson | C. Papoutsi | R. Foy | R. Boaden | J. Rycroft-Malone | E. Howarth | L. Moore | M. Campbell | R. Raine | N. Fulop | R. McDonald | L. Goulding | Z. Or | C. Gillies | R. Newhouse | R. Boaden | Kelly J. Devers | H. Barratt | Lucy Goulding | G. Bevan | E. Gibbons | S. Prady | Robin P. Newhouse | Gywn Bevan | Elizabeth Gibbons | Clare L. Gillies | Emma Howarth | R. Raine | Emma Howarth | Nick Freemantle | Ray Fitzpatrick | Jeremy M Grimshaw | Richard Grieve | Merrick Zwarenstein | Richard J. Lilford | Graham Moore | Robbie Foy | L. Fallowfield | A. O’Cathain | Naomi Fulop | Marion K Campbell | Gywn Bevan | Nick Black | Peter Bower | Marion K Campbell | Jean-Louis Denis | Lesley Fallowfield | Clare L. Gillies | Lucy Goulding | Emma Howarth | Ruth McDonald | Laurence Moore | Robin P. Newhouse | Zeynep Or | J. Rycroft-Malone | Simon Turner | Samuel I Watson | Samuel I Watson | J. M. Grimshaw

[1]  M. Petticrew,et al.  Developing and evaluating complex interventions: the new Medical Research Council guidance , 2008, BMJ : British Medical Journal.

[2]  R. Brook,et al.  Using the Knowledge Base of Health Services Research to Redefine Health Care Systems , 2015, Journal of General Internal Medicine.

[3]  N. Freemantle,et al.  Making inferences on treatment effects from real world data: propensity scores, confounding by indication, and other perils for the unwary in observational research , 2013, BMJ.

[4]  P. Kontos,et al.  Implementation Science BioMed Central Debate , 2009 .

[5]  D. Ashby,et al.  Sample size for cluster randomized trials: effect of coefficient of variation of cluster size and analysis method. , 2006, International journal of epidemiology.

[6]  M. Harrison,et al.  Building capacity for a transformation initiative: system redesign at Denver Health. , 2009, Health care management review.

[7]  J. Sekhon,et al.  Genetic Matching for Estimating Causal Effects: A General Multivariate Matching Method for Achieving Balance in Observational Studies , 2006, Review of Economics and Statistics.

[8]  L. Lorenzoni,et al.  Informing policy makers about future health spending: a comparative analysis of forecasting methods in OECD countries. , 2012, Health policy.

[9]  Rosalind Raine,et al.  Social variations in access to hospital care for patients with colorectal, breast, and lung cancer between 1999 and 2006: retrospective analysis of hospital episode statistics , 2010, BMJ : British Medical Journal.

[10]  N. Best,et al.  Adjusting for Selection Effects in Epidemiologic Studies: Why Sensitivity Analysis is the Only “Solution” , 2011, Epidemiology.

[11]  Robert E. Goodin,et al.  The Oxford Handbook of Contextual Political Analysis , 2006 .

[12]  N. Black,et al.  Relationship Between Patient-reported Outcomes of Elective Surgery and Hospital and Consultant Volume , 2015, Medical care.

[13]  R. Newhouse,et al.  The Patient-Centered Outcomes Research Institute: Research Done Differently , 2015, Nursing research.

[14]  Sam Porter,et al.  On the problems of mixing RCTs with qualitative research: the case of the MRC framework for the evaluation of complex healthcare interventions , 2010 .

[15]  R. Hirasing,et al.  Effects of nurse home visitation on cigarette smoking, pregnancy outcomes and breastfeeding: a randomized controlled trial. , 2014, Midwifery.

[16]  J. Newhouse A Summary of the Rand Health Insurance Study * , 1982, Annals of the New York Academy of Sciences.

[17]  C. Normand,et al.  Changing organisations: a study of the context and processes of mergers of health care providers in England. , 2005, Social science & medicine.

[18]  T. Greenhalgh,et al.  Realist review - a new method of systematic review designed for complex policy interventions , 2005, Journal of health services research & policy.

[19]  Karl Claxton,et al.  Methods for the estimation of the National Institute for Health and Care Excellence cost-effectiveness threshold. , 2015, Health technology assessment.

[20]  A. Best,et al.  Systems thinking, knowledge and action: towards better models and methods , 2010 .

[21]  Jean Adams,et al.  The role of receipt and timeliness of treatment in socioeconomic inequalities in lung cancer survival: population-based, data-linkage study , 2014, Thorax.

[22]  Luke W. Miratrix,et al.  Adjusting treatment effect estimates by post‐stratification in randomized experiments , 2013 .

[23]  M. Soljak,et al.  NHS Health Checks , 2014, The Journal of ambulatory care management.

[24]  J. Forder,et al.  Using cost-effectiveness estimates from survey data to guide commissioning: an application to home care. , 2014, Health economics.

[25]  T. Wykes,et al.  Impact of patient involvement in mental health research: longitudinal study , 2013, British Journal of Psychiatry.

[26]  Gianluca Baio,et al.  Stepped wedge randomised controlled trials: systematic review of studies published between 2010 and 2014 , 2015, Trials.

[27]  P. J. Chilton,et al.  Evaluation of a predevelopment service delivery intervention: an application to improve clinical handovers , 2012, BMJ quality & safety.

[28]  David Richards,et al.  Complex Interventions in Health : An overview of research methods , 2015 .

[29]  T. Hoffmann,et al.  Poor description of non-pharmacological interventions: analysis of consecutive sample of randomised trials , 2013, BMJ : British Medical Journal.

[30]  Peter J Pronovost,et al.  Reality check for checklists , 2009, The Lancet.

[31]  David Moher,et al.  Reporting of patient-reported outcomes in randomized trials: the CONSORT PRO extension. , 2013, JAMA.

[32]  Allan Hackshaw,et al.  Effects of evidence-based strategies to reduce the socioeconomic gradient of uptake in the English NHS Bowel Cancer Screening Programme (ASCEND): four cluster-randomised controlled trials , 2016, The Lancet.

[33]  Jon Nicholl,et al.  Case-mix adjustment in non-randomised observational evaluations: the constant risk fallacy , 2007, Journal of Epidemiology & Community Health.

[34]  G. Schiff,et al.  Free for All? Lessons from the RAND Health Insurance Experiment , 2001 .

[35]  Susan Michie,et al.  Developing and Evaluating Complex Interventions , 2015 .

[36]  I. Higginson,et al.  Implementing patient reported outcome measures (PROMs) in palliative care - users' cry for help , 2011, Health and quality of life outcomes.

[37]  E. Goyder,et al.  Inequalities in breast cancer stage at diagnosis in the Trent region, and implications for the NHS Breast Screening Programme. , 2009, Journal of public health.

[38]  Mary Dixon-Woods,et al.  What to expect when you're evaluating healthcare improvement: a concordat approach to managing collaboration and uncomfortable realities , 2015, BMJ Quality & Safety.

[39]  J. Grimshaw,et al.  Looking inside the black box: results of a theory-based process evaluation exploring the results of a randomized controlled trial of printed educational messages to increase primary care physicians' diabetic retinopathy referrals [Trial registration number ISRCTN72772651]. , 2014, Implementation science : IS.

[40]  Brian S Mittman,et al.  Creating the Evidence Base for Quality Improvement Collaboratives , 2004, Annals of Internal Medicine.

[41]  Kyle Possemato,et al.  Developing a measure of provider adherence to improve the implementation of behavioral health services in primary care: a Delphi study , 2013, Implementation Science.

[42]  Karen Golden-Biddle,et al.  Identity Struggles in Merging Organizations , 2012 .

[43]  D. Albarracín,et al.  A test of major assumptions about behavior change: a comprehensive look at the effects of passive and active HIV-prevention interventions since the beginning of the epidemic. , 2005, Psychological bulletin.

[44]  M. Johnston,et al.  From Theory to Intervention: Mapping Theoretically Derived Behavioural Determinants to Behaviour Change Techniques , 2008 .

[45]  N. Black,et al.  Integrating the use of patient-reported outcomes for both clinical practice and performance measurement: views of experts from 3 countries. , 2014, The Milbank quarterly.

[46]  J. Woodward,et al.  Saving the phenomena , 1988 .

[47]  A. D. de Bont,et al.  The contribution of qualitative research to the development of tailor-made community-based interventions in primary care: a review. , 2010, European journal of public health.

[48]  J Coste,et al.  Methodological approaches to shortening composite measurement scales. , 1997, Journal of clinical epidemiology.

[49]  M. Petticrew,et al.  Using natural experiments to evaluate population health interventions: new Medical Research Council guidance , 2012, Journal of Epidemiology & Community Health.

[50]  Natalie J Gould,et al.  Application of theory to enhance audit and feedback interventions to increase the uptake of evidence-based transfusion practice: an intervention development protocol , 2014, Implementation Science.

[51]  P. Pronovost,et al.  Strategies to Improve Patient Safety: The Evidence Base Matures , 2013, Annals of Internal Medicine.

[52]  E. Pujade-Lauraine,et al.  Toxicity and quality of life outcomes in ovarian cancer patients participating in randomized controlled trials , 2011, Supportive Care in Cancer.

[53]  C. Dowrick,et al.  Implementation Science Development of a Theory of Implementation and Integration: Normalization Process Theory , 2022 .

[54]  N. Laird,et al.  Meta-analysis in clinical trials. , 1986, Controlled clinical trials.

[55]  P. Shekelle,et al.  The effectiveness of computerized order entry at reducing preventable adverse drug events and medication errors in hospital settings: a systematic review and meta-analysis , 2014, Systematic Reviews.

[56]  P. Nilsen Making sense of implementation theories, models and frameworks , 2015, Implementation Science.

[57]  C. Henderson,et al.  Preventing child abuse and neglect: a randomized trial of nurse home visitation. , 1986, Pediatrics.

[58]  K. Tapper,et al.  The impact of school fruit tuck shops and school food policies on children’s fruit consumption: a cluster randomised trial of schools in deprived areas , 2008, Journal of Epidemiology & Community Health.

[59]  B. Pitt,et al.  Eplerenone , a Selective Aldosterone Blocker , in Patients with Left Ventricular Dysfunction after Myocardial Infarction , 2003 .

[60]  C. McKevitt,et al.  Lessons for major system change: centralization of stroke services in two metropolitan areas of England , 2016, Journal of health services research & policy.

[61]  Josefien van Olmen,et al.  Is realist evaluation keeping its promise? A review of published empirical studies in the field of health systems research , 2012 .

[62]  David Feeny,et al.  Training clinicians in how to use patient-reported outcome measures in routine clinical practice , 2015, Quality of Life Research.

[63]  Jean-Louis Denis,et al.  Beyond evidence: the micropolitics of improvement , 2011, Quality and Safety in Health Care.

[64]  K. Walters,et al.  Explaining variation in referral from primary to secondary care: cohort study , 2010, BMJ : British Medical Journal.

[65]  D. Forman,et al.  Socioeconomic background in relation to stage at diagnosis, treatment and survival in women with breast cancer , 2007, British Journal of Cancer.

[66]  Monitoring inequalities in the National Chlamydia Screening Programme in England: added value of ACORN, a commercial geodemographic classification tool. , 2009, Sexual health.

[67]  Manuel Gomes,et al.  Addressing Missing Data in Patient‐Reported Outcome Measures (PROMS): Implications for the Use of PROMS for Comparing Provider Performance , 2015, Health economics.

[68]  David J Spiegelhalter,et al.  Bias modelling in evidence synthesis , 2009, Journal of the Royal Statistical Society. Series A,.

[69]  Richard Emsley,et al.  Ian R White interventions Mediation and moderation of treatment effects in randomised controlled trials of complex , 2010 .

[70]  Richard Gray,et al.  Comparing results from long and short form versions of the Parkinson's disease questionnaire in a longitudinal study. , 2015, Parkinsonism & related disorders.

[71]  R. D'Agostino Adjustment Methods: Propensity Score Methods for Bias Reduction in the Comparison of a Treatment to a Non‐Randomized Control Group , 2005 .

[72]  Jenny Hewison,et al.  Maximising the value of combining qualitative research and randomised controlled trials in health research: the QUAlitative Research in Trials (QUART) study--a mixed methods study. , 2014, Health technology assessment.

[73]  J. Hippisley-Cox,et al.  Inequalities in access to coronary angiography and revascularisation: the association of deprivation and location of primary care services. , 2000, The British journal of general practice : the journal of the Royal College of General Practitioners.

[74]  M. Dixon-Woods,et al.  What counts? An ethnographic study of infection data reported to a patient safety program. , 2012, The Milbank quarterly.

[75]  H. Doll,et al.  Change in health status in long-term conditions over a one year period: a cohort survey using patient-reported outcome measures , 2014, Health and Quality of Life Outcomes.

[76]  R. Fitzpatrick,et al.  Impact of patient-reported outcome measures on routine practice: a structured review. , 2006, Journal of evaluation in clinical practice.

[77]  J. Forder Long-term care and hospital utilisation by older people: an analysis of substitution rates. , 2009, Health economics.

[78]  E. Bateman,et al.  Effect of educational outreach to nurses on tuberculosis case detection and primary care of respiratory illness: pragmatic cluster randomised controlled trial , 2005, BMJ : British Medical Journal.

[79]  A. Mattioli,et al.  Collaborative meta-analysis of randomised trials of antiplatelet therapy for prevention of death, myocardial infarction, and stroke in high risk patients , 2002, BMJ : British Medical Journal.

[80]  Carl Heneghan,et al.  What is missing from descriptions of treatment in trials and reviews? , 2008, BMJ : British Medical Journal.

[81]  L. McLeod,et al.  ISOQOL recommends minimum standards for patient-reported outcome measures used in patient-centered outcomes and comparative effectiveness research , 2013, Quality of Life Research.

[82]  C. McKevitt,et al.  Stop Stroke: development of an innovative intervention to improve risk factor management after stroke. , 2008, Patient education and counseling.

[83]  Anita Kothari,et al.  Using realist evaluation to open the black box of knowledge translation: a state-of-the-art review , 2014, Implementation Science.

[84]  Henna Hasson,et al.  Fidelity and moderating factors in complex interventions: a case study of a continuum of care program for frail elderly people in health and social care , 2012, Implementation Science.

[85]  Susan Michie,et al.  Specifying and reporting complex behaviour change interventions: the need for a scientific method , 2009, Implementation science : IS.

[86]  S. Mulvagh,et al.  The Challenges of Prevention, Diagnosis and Treatment of Ischemic Heart Disease in Women , 2015, Cardiovascular Drugs and Therapy.

[87]  Matt Sutton,et al.  Examination of the Synthetic Control Method for Evaluating Health Policies with Multiple Treated Units , 2015, Health economics.

[88]  K. Brazil,et al.  Revisiting the Quantitative-Qualitative Debate: Implications for Mixed-Methods Research , 2002, Quality & quantity.

[89]  R Brian Haynes,et al.  Features of effective computerised clinical decision support systems: meta-regression of 162 randomised trials , 2013, BMJ : British Medical Journal.

[90]  R. Lyons,et al.  Impacts of the Primary School Free Breakfast Initiative on socio-economic inequalities in breakfast consumption among 9–11-year-old schoolchildren in Wales , 2013, Public Health Nutrition.

[91]  M. Dixon-Woods,et al.  Did a quality improvement collaborative make stroke care better? A cluster randomized trial , 2014, Implementation Science.

[92]  R. Yin Validity and generalization in future case study evaluations , 2013 .

[93]  C. Wolfe,et al.  Point-of-Care Cluster Randomized Trial in Stroke Secondary Prevention Using Electronic Health Records , 2014, Stroke.

[94]  R. Lilford,et al.  Bmc Medical Research Methodology Open Access the Stepped Wedge Trial Design: a Systematic Review , 2022 .

[95]  N. Black,et al.  Clinicians’ and patients’ views of metrics of change derived from patient reported outcome measures (PROMs) for comparing providers’ performance of surgery , 2012, BMC Health Services Research.

[96]  Nigel Rice,et al.  Econometric Evaluation of Health Policies , 2011 .

[97]  N Freemantle,et al.  Weekend hospitalization and additional risk of death: An analysis of inpatient data , 2012, Journal of the Royal Society of Medicine.

[98]  A. Lockett,et al.  From what we know to what we do: lessons learned from the translational CLAHRC initiative in England , 2013, Journal of health services research & policy.

[99]  Peter Mendel,et al.  Implementation of the CALM intervention for anxiety disorders: a qualitative study , 2012, Implementation Science.

[100]  Z. Eisikovits,et al.  Physician as partner or salesman? Shared decision-making in real-time encounters. , 2009, Social science & medicine.

[101]  Andrew Booth,et al.  Implementation Science BioMed Central Debate A conceptual framework for implementation fidelity , 2007 .

[102]  Anne W. Riley,et al.  Pediatric patient-reported outcome instruments for research to support medical product labeling: report of the ISPOR PRO good research practices for the assessment of children and adolescents task force. , 2013, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[103]  Keith R Abrams,et al.  Factors affecting uptake of childhood immunisation: a Bayesian synthesis of qualitative and quantitative evidence , 2002, The Lancet.

[104]  Jill J. Francis,et al.  Designing theoretically-informed implementation interventions , 2006, Implementation science : IS.

[105]  C. Victora,et al.  Explaining trends in inequities: evidence from Brazilian child health studies , 2000, The Lancet.

[106]  J. Ioannidis,et al.  Comparison of evidence of treatment effects in randomized and nonrandomized studies. , 2001, JAMA.

[107]  Ginny Brunton,et al.  Using qualitative comparative analysis (QCA) in systematic reviews of complex interventions: a worked example , 2014, Systematic Reviews.

[108]  J. Grimshaw,et al.  The Ottawa Statement on the Ethical Design and Conduct of Cluster Randomized Trials , 2012, PLoS medicine.

[109]  C. von Wagner,et al.  Outcomes of the Bowel Cancer Screening Programme (BCSP) in England after the first 1 million tests , 2010, Gut.

[110]  Brian Williams,et al.  Intervention description is not enough: evidence from an in-depth multiple case study on the untold role and impact of context in randomised controlled trials of seven complex interventions , 2012, Trials.

[111]  J. Grimshaw,et al.  The translation research in a dental setting (TRiaDS) programme protocol , 2010, Implementation science : IS.

[112]  Raimund Erbel,et al.  Perceived job insecurity as a risk factor for incident coronary heart disease: systematic review and meta-analysis , 2013, BMJ.

[113]  P. Pronovost,et al.  Did hospital engagement networks actually improve care? , 2014, The New England journal of medicine.

[114]  C. Sherbourne,et al.  The Health Insurance Experiment A Classic RAND Study Speaks to the Current Health Care Reform Debate , 2006 .

[115]  Trisha Greenhalgh,et al.  How do you modernize a health service? A realist evaluation of whole-scale transformation in london. , 2009, The Milbank quarterly.

[116]  Catherine Sherrington,et al.  Cluster randomised trial of a targeted multifactorial intervention to prevent falls among older people in hospital , 2008, BMJ : British Medical Journal.

[117]  Bruce Guthrie,et al.  Process evaluations for cluster-randomised trials of complex interventions: a proposed framework for design and reporting , 2013, Trials.

[118]  Karla Hemming,et al.  Large scale organisational intervention to improve patient safety in four UK hospitals: mixed method evaluation , 2011, BMJ : British Medical Journal.

[119]  J. McMurray,et al.  Eplerenone in patients with systolic heart failure and mild symptoms. , 2011, The New England journal of medicine.

[120]  Joy Adamson,et al.  The opportunities and challenges of pragmatic point-of-care randomised trials using routinely collected electronic records: evaluations of two exemplar trials. , 2014, Health technology assessment.

[121]  G. Velikova,et al.  Use of patient-reported outcomes to measure symptoms and health related quality of life in the clinic. , 2015, Gynecologic oncology.

[122]  M. Laudicella,et al.  OP44 A framework for monitoring nhs equity performance – small area analysis of national administrative data from 2004/5 to 2011/12 , 2015, Journal of Epidemiology & Community Health.

[123]  A. O’Cathain,et al.  Process evaluation of complex interventions: Medical Research Council guidance , 2015, BMJ : British Medical Journal.

[124]  G. Guyatt,et al.  The impact of measuring patient-reported outcomes in clinical practice: a systematic review of the literature , 2008, Quality of Life Research.

[125]  S. Duffy,et al.  Strategy for power calculation for interactions: application to a trial of interventions to improve uptake of bowel cancer screening. , 2012, Contemporary clinical trials.

[126]  Jessica Datta,et al.  Challenges to evaluating complex interventions: a content analysis of published papers , 2013, BMC Public Health.

[127]  Ellwood Pm,et al.  Shattuck Lecture--outcomes management. A technology of patient experience. 1988. , 1997, Archives of pathology & laboratory medicine.

[128]  P. Glasziou,et al.  Systematic review automation technologies , 2014, Systematic Reviews.

[129]  G. Moore,et al.  Free healthy breakfasts in primary schools: a cluster randomised controlled trial of a policy intervention in Wales, UK , 2010, Public Health Nutrition.

[130]  Fernando Fernandez-Llimos,et al.  A systematic review of implementation frameworks of innovations in healthcare and resulting generic implementation framework , 2015, Health Research Policy and Systems.

[131]  N. Mays,et al.  RCTs: How compatible are they with policy-making? , 2015 .

[132]  P. Sandercock,et al.  Framework for design and evaluation of complex interventions to improve health , 2000, BMJ : British Medical Journal.

[133]  C. von Wagner,et al.  Uptake of Bowel Scope (Flexible Sigmoidoscopy) Screening in the English National Programme: the first 14 months , 2016, Journal of medical screening.

[134]  Matt Sutton,et al.  Reduced mortality with hospital pay for performance in England. , 2012, The New England journal of medicine.

[135]  A. O’Cathain,et al.  Impact of the urgent care telephone service NHS 111 pilot sites: a controlled before and after study , 2013, BMJ Open.

[136]  Shaun Treweek,et al.  Methods to improve recruitment to randomised controlled trials: Cochrane systematic review and meta-analysis , 2013, BMJ Open.

[137]  I. Tannock,et al.  Bias in reporting of end points of efficacy and toxicity in randomized, clinical trials for women with breast cancer. , 2013, Annals of oncology : official journal of the European Society for Medical Oncology.

[138]  J. Ioannidis,et al.  Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials , 2011, BMJ : British Medical Journal.

[139]  Susan Michie,et al.  Demystifying theory and its use in improvement , 2015, BMJ quality & safety.

[140]  J. Donovan,et al.  Clear obstacles and hidden challenges: understanding recruiter perspectives in six pragmatic randomised controlled trials , 2014, Trials.

[141]  L. Lipsitz,et al.  Understanding health care as a complex system: the foundation for unintended consequences. , 2012, JAMA.

[142]  Conor Mayo-Wilson,et al.  The Limits of Piecemeal Causal Inference , 2014, The British Journal for the Philosophy of Science.

[143]  David A Chambers,et al.  Bridging research and practice: models for dissemination and implementation research. , 2012, American journal of preventive medicine.

[144]  Alan Shiell,et al.  Complex interventions or complex systems? Implications for health economic evaluation , 2008, BMJ : British Medical Journal.

[145]  Alain Mayhew,et al.  Decisions about lumping vs. splitting of the scope of systematic reviews of complex interventions are not well justified: a case study in systematic reviews of health care professional reminders. , 2012, Journal of clinical epidemiology.

[146]  A. Majeed,et al.  Primary care factors associated with cervical screening coverage in England. , 2012, Journal of public health.

[147]  John E. Wennberg,et al.  Geographic Variation in Expenditures for Physicians' Services in the United States , 1993 .

[148]  P. Donnan,et al.  The PRECIS-2 tool: designing trials that are fit for purpose , 2015, BMJ : British Medical Journal.

[149]  P. Fayers,et al.  Determinants of the intracluster correlation coefficient in cluster randomized trials: the case of implementation research , 2005, Clinical trials.

[150]  T. Greenhalgh,et al.  Realist RCTs of complex interventions - an oxymoron. , 2013, Social science & medicine.

[151]  M. Wolpert Uses and Abuses of Patient Reported Outcome Measures (PROMs): Potential Iatrogenic Impact of PROMs Implementation and How It Can Be Mitigated , 2013, Administration and Policy in Mental Health and Mental Health Services Research.

[152]  J. Forder,et al.  Personalization in the health care system: do personal health budgets have an impact on outcomes and cost? , 2013, Journal of health services research & policy.

[153]  P. Bower,et al.  Effect of telehealth on quality of life and psychological outcomes over 12 months (Whole Systems Demonstrator telehealth questionnaire study): nested study of patient reported outcomes in a pragmatic, cluster randomised controlled trial , 2013, BMJ : British Medical Journal.

[154]  J. Grimshaw,et al.  Developing theory-informed behaviour change interventions to implement evidence into practice: a systematic approach using the Theoretical Domains Framework , 2012, Implementation Science.

[155]  G. Guyatt,et al.  Clinical impact versus factor analysis for quality of life questionnaire construction. , 1997, Journal of clinical epidemiology.

[156]  R. Glasgow,et al.  Why don't we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. , 2003, American journal of public health.

[157]  Joanne Greenhalgh,et al.  Surgeon’s experiences of receiving peer benchmarked feedback using patient-reported outcome measures: a qualitative study , 2014, Implementation Science.

[158]  B. Pitt,et al.  The effect of spironolactone on morbidity and mortality in patients with severe heart failure. Randomized Aldactone Evaluation Study Investigators. , 1999, The New England journal of medicine.

[159]  Jo Rycroft-Malone,et al.  The PARIHS framework--a framework for guiding the implementation of evidence-based practice. , 2004, Journal of nursing care quality.

[160]  M. Kenward,et al.  Differential dropout and bias in randomised controlled trials: when it matters and when it may not , 2013, BMJ.

[161]  L. Siciliani,et al.  Testing the Bed-Blocking Hypothesis: Does Nursing and Care Home Supply Reduce Delayed Hospital Discharges? , 2015, Health economics.

[162]  J. Newhouse,et al.  Econometrics in outcomes research: the use of instrumental variables. , 1998, Annual review of public health.

[163]  C. Terwee,et al.  Rating the methodological quality in systematic reviews of studies on measurement properties: a scoring system for the COSMIN checklist , 2011, Quality of Life Research.

[164]  Roland R. Ramsahai,et al.  From sample average treatment effect to population average treatment effect on the treated: combining experimental with observational studies to estimate population treatment effects , 2015 .

[165]  C. Henderson,et al.  Effect of prenatal and infancy home visitation by nurses on pregnancy outcomes, childhood injuries, and repeated childbearing. A randomized controlled trial. , 1997, JAMA.

[166]  G. Baio,et al.  Five questions to consider before conducting a stepped wedge trial , 2015, Trials.

[167]  M. Bloor,et al.  Process evaluation: the new miracle ingredient in public health research? , 2010 .

[168]  A. Tennant,et al.  A systematic review of generic multidimensional patient-reported outcome measures for children, part I: descriptive characteristics. , 2015, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[169]  Kate Flemming,et al.  Electronic searching to locate qualitative research: evaluation of three strategies. , 2007, Journal of advanced nursing.

[170]  David J Spiegelhalter,et al.  Bayesian approaches to multiple sources of evidence and uncertainty in complex cost‐effectiveness modelling , 2003, Statistics in medicine.

[171]  M. Whitehead,et al.  The impact of NHS resource allocation policy on health inequalities in England 2001-11: longitudinal ecological study , 2014, BMJ : British Medical Journal.

[172]  L. Bero,et al.  Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials. , 2014, The Cochrane database of systematic reviews.

[173]  R. Fitzpatrick,et al.  Patient reports of the outcomes of treatment: a structured review of approaches , 2014, Health and Quality of Life Outcomes.

[174]  S. James Press,et al.  Subjective and Objective Bayesian Statistics , 2002 .

[175]  N. Black,et al.  Estimating recruitment rates for routine use of patient reported outcome measures and the impact on provider comparisons , 2014, BMC Health Services Research.

[176]  Ronald C Plotnikoff,et al.  Factors influencing the adoption of an innovation: An examination of the uptake of the Canadian Heart Health Kit (HHK) , 2008, Implementation science : IS.

[177]  J. Hayden,et al.  Methods of a multi-faceted rapid knowledge synthesis project to inform the implementation of a new health service model: Collaborative Emergency Centres , 2015, Systematic Reviews.

[178]  M. Dixon-Woods,et al.  Can an electronic prescribing system detect doctors who are more likely to make a serious prescribing error , 2011 .

[179]  G. Stoddart,et al.  Controlling health expenditures--the Canadian reality. , 1989, The New England journal of medicine.

[180]  J Golding,et al.  Environment of infants during sleep and risk of the sudden infant death syndrome: results of 1993-5 case-control study for confidential inquiry into stillbirths and deaths in infancy , 1996, BMJ.

[181]  Sylvia J. Hysong,et al.  Meta-Analysis: Audit and Feedback Features Impact Effectiveness on Care Quality , 2009, Medical care.

[182]  R. Grol,et al.  Factors influencing success in quality-improvement collaboratives: development and psychometric testing of an instrument , 2010, Implementation science : IS.

[183]  H. Krumholz Big data and new knowledge in medicine: the thinking, training, and tools needed for a learning health system. , 2014, Health affairs.

[184]  N. Freemantle,et al.  Fatal venous thromboembolism associated with hospital admission: a cohort study to assess the impact of a national risk assessment target , 2013, Heart.

[185]  P. J. Chilton,et al.  Bayesian Cohort and Cross-Sectional Analyses of the PINCER Trial: A Pharmacist-Led Intervention to Reduce Medication Errors in Primary Care , 2012, PloS one.

[186]  S. Greenland An introduction To instrumental variables for epidemiologists , 2000, International journal of epidemiology.

[187]  David Grembowski,et al.  Emerging lessons from regional and state innovation in value-based payment reform: balancing collaboration and disruptive innovation. , 2014, The Milbank quarterly.

[188]  N. Mays,et al.  The Multiple Purposes of Policy Piloting and Their Consequences: Three Examples from National Health and Social Care Policy in England , 2014, Journal of Social Policy.

[189]  C. Haraden,et al.  Scotland's successful national approach to improving patient safety in acute care. , 2011, Health affairs.

[190]  A. Kluger,et al.  The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. , 1996 .

[191]  J. Katz,et al.  Patient preferences and health disparities. , 2001, JAMA.

[192]  J. Brazier,et al.  Outcomes of social care for adults: developing a preference-weighted measure. , 2012, Health technology assessment.

[193]  J. Ovretveit,et al.  Learning from large-scale quality improvement through comparisons. , 2012, International journal for quality in health care : journal of the International Society for Quality in Health Care.

[194]  Denise M. Rousseau,et al.  Reinforcing the Micro/Macro Bridge: Organizational Thinking and Pluralistic Vehicles , 2011 .

[195]  A. Laupacis,et al.  Printed educational messages aimed at family practitioners fail to increase retinal screening among their patients with diabetes: a pragmatic cluster randomized controlled trial [ISRCTN72772651] , 2014, Implementation Science.

[196]  Alan Shiell,et al.  Complex interventions: how “out of control” can a randomised controlled trial be? , 2004, BMJ : British Medical Journal.

[197]  G. Suresh Neonatal MRI and neurodevelopmental outcomes. , 2006, The New England journal of medicine.

[198]  Sue Newell,et al.  Networked innovation in the health sector: comparative qualitative study of the role of Collaborations for Leadership in Applied Health Research and Care in translating research into practice , 2014 .

[199]  K. Wolf-Ostermann,et al.  Quality of life ratings in dementia care – a cross-sectional study to identify factors associated with proxy-ratings , 2014, Health and Quality of Life Outcomes.

[200]  Jon G Ayres,et al.  Secondhand smoke levels in Scottish pubs: the effect of smoke-free legislation , 2007, Tobacco Control.

[201]  Hadi Kharrazi,et al.  Measure once, cut twice d adding patient-reported outcome measures to the electronic health record for comparative effectiveness research , 2013 .

[202]  Tina Miller,et al.  Finding qualitative research: an evaluation of search strategies , 2004, BMC medical research methodology.

[203]  Paula R. Williamson,et al.  Choosing Important Health Outcomes for Comparative Effectiveness Research: A Systematic Review , 2014, PloS one.

[204]  K. Kroenke,et al.  Pragmatic characteristics of patient-reported outcome measures are important for use in clinical practice. , 2015, Journal of clinical epidemiology.

[205]  Peter J Pronovost,et al.  Explaining Michigan: developing an ex post theory of a quality improvement program. , 2011, The Milbank quarterly.

[206]  Joshua D. Angrist,et al.  Mostly Harmless Econometrics: An Empiricist's Companion , 2008 .

[207]  J. Achten,et al.  Outcome assessment after hip fracture , 2014, Bone & joint research.

[208]  J. Sterne,et al.  The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials , 2011, BMJ : British Medical Journal.

[209]  Tony Cornford,et al.  Building a house on shifting sand: methodological considerations when evaluating the implementation and adoption of national electronic health record systems , 2012, BMC Health Services Research.

[210]  R. Grieve,et al.  An Approach to Assess Generalizability in Comparative Effectiveness Research , 2015, Medical decision making : an international journal of the Society for Medical Decision Making.

[211]  Alan Shiell,et al.  Theorising Interventions as Events in Systems , 2009, American journal of community psychology.

[212]  Gerry Richardson,et al.  Effectiveness of a nurse-led intensive home-visitation programme for first-time teenage mothers (Building Blocks): a pragmatic randomised controlled trial , 2016, The Lancet.

[213]  M. Dixon-Woods,et al.  How collaborative are quality improvement collaboratives: a qualitative study in stroke care , 2014, Implementation Science.

[214]  D. Olds Prenatal and Infancy Home Visiting by Nurses: From Randomized Trials to Community Replication , 2002, Prevention Science.

[215]  M. Sutton,et al.  A qualitative and quantitative evaluation of the Advancing Quality pay-for-performance programme in the NHS North West , 2015 .

[216]  H. Doll,et al.  The routine collection of patient-reported outcome measures (PROMs) for long-term conditions in primary care: a cohort survey , 2014, BMJ Open.

[217]  Nick Black,et al.  Relationship between patient reported experience (PREMs) and patient reported outcomes (PROMs) in elective surgery , 2014, BMJ quality & safety.

[218]  Charles Heckscher,et al.  Perspective - Professional Work: The Emergence of Collaborative Community , 2008, Organ. Sci..

[219]  C. Hertzman,et al.  The 20-year experiment: accounting for, explaining, and evaluating health care cost containment in Canada and the United States. , 1991, Annual review of public health.

[220]  U. Dirnagl,et al.  Biomedical research: increasing value, reducing waste , 2014, The Lancet.

[221]  Katherine Baicker,et al.  Medicaid Increases Emergency-Department Use: Evidence from Oregon's Health Insurance Experiment , 2014, Science.

[222]  T. Blackman,et al.  Qualitative Comparative Analysis and Health Inequalities: Investigating Reasons for Differential Progress with Narrowing Local Gaps in Mortality , 2010, Journal of Social Policy.

[223]  Jo Rycroft-Malone,et al.  An implementation research agenda , 2009, Implementation science : IS.

[224]  Christina Keller,et al.  Implementing organisation and management innovations in Swedish healthcare: lessons from a comparison of 12 cases. , 2012, Journal of health organization and management.

[225]  M J Buxton,et al.  Evaluating patient-based outcome measures for use in clinical trials. , 1998, Health technology assessment.

[226]  Gaetano Rocco,et al.  Symptomatic toxicities experienced during anticancer treatment: agreement between patient and physician reporting in three randomized trials. , 2015, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.

[227]  Elliott S Fisher,et al.  Analysis of observational studies in the presence of treatment selection bias: effects of invasive cardiac management on AMI survival using propensity score and instrumental variable methods. , 2007, JAMA.

[228]  N. Mays,et al.  Policy experiments: Investigating effectiveness or confirming direction? , 2015 .

[229]  J. Hendy,et al.  Effect of telehealth on use of secondary care and mortality: findings from the Whole System Demonstrator cluster randomised trial , 2012, BMJ : British Medical Journal.

[230]  R J Lilford,et al.  The stepped wedge cluster randomised trial: rationale, design, analysis, and reporting , 2015, BMJ : British Medical Journal.

[231]  N. Black,et al.  Factors associated with non-response in routine use of patient reported outcome measures after elective surgery in England , 2012, Health and Quality of Life Outcomes.

[232]  Julie Polisena,et al.  Rapid review programs to support health care and policy decision making: a descriptive analysis of processes and methods , 2015, Systematic Reviews.

[233]  Karla Hemming,et al.  Evaluating policy and service interventions: framework to guide selection and interpretation of study end points , 2010, BMJ : British Medical Journal.

[234]  Aki Tsuchiya,et al.  A review of studies mapping (or cross walking) non-preference based measures of health to generic preference-based measures , 2010, The European Journal of Health Economics.

[235]  C. Dowrick,et al.  Complex interventions , 2022, International Review of Sport and Exercise Psychology.

[236]  M. Sutton Vertical and horizontal aspects of socio-economic inequity in general practitioner contacts in Scotland. , 2002, Health economics.

[237]  J. De Maeseneer,et al.  Socio-economic status of the patient and doctor-patient communication: does it make a difference? , 2005, Patient education and counseling.

[238]  J. Furler,et al.  Bridging the Gap: using an interrupted time series design to evaluate systems reform addressing refugee maternal and child health inequalities , 2015, Implementation Science.

[239]  B. Pitt,et al.  The Effect of Spironolactone on Morbidity and Mortality in Patients with Severe Heart Failure , 2000 .

[240]  Jon Nicholl,et al.  A reinvestigation of recruitment to randomised, controlled, multicenter trials: a review of trials funded by two UK funding agencies , 2013, Trials.

[241]  J. Newhouse,et al.  Income, insurance, and technology: why does health spending outpace economic growth? , 2009, Health affairs.

[242]  J. Vaupel,et al.  Ageing populations: the challenges ahead , 2009, The Lancet.

[243]  Susan Michie,et al.  No more ‘business as usual’ with audit and feedback interventions: towards an agenda for a reinvigorated intervention , 2014, Implementation Science.

[244]  R. Fitzpatrick,et al.  The Outcomes and Experiences Questionnaire: development and validation , 2014, Patient related outcome measures.

[245]  T. Falleti,et al.  Context and Causal Mechanisms in Political Analysis , 2009 .

[246]  C. Abraham,et al.  Making psychological theory useful for implementing evidence based practice: a consensus approach , 2005, Quality and Safety in Health Care.

[247]  Penelope Hawe,et al.  Lessons from complex interventions to improve health. , 2015, Annual review of public health.

[248]  C. May Towards a general theory of implementation , 2013, Implementation Science.

[249]  Anthony O'Hagan,et al.  Eliciting expert beliefs in substantial practical applications , 1998 .

[250]  David Cella,et al.  The Role of Technical Advances in the Adoption and Integration of Patient-reported Outcomes in Clinical Care , 2015, Medical care.

[251]  Role of politics in understanding complex, messy health systems: an essay by David J Hunter , 2015, BMJ : British Medical Journal.

[252]  J. Wyatt,et al.  Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide , 2014, BMJ : British Medical Journal.

[253]  Jenny Moberg,et al.  A brief history of the cluster randomised trial design , 2015, Journal of the Royal Society of Medicine.

[254]  David J. Spiegelhalter,et al.  Bayesian graphical modelling: a case‐study in monitoring health outcomes , 2002 .

[255]  K. Bhaskaran,et al.  Data Resource Profile: Clinical Practice Research Datalink (CPRD) , 2015, International journal of epidemiology.

[256]  Jeremy M. Grimshaw,et al.  Sample size calculator for cluster randomized trials , 2004, Comput. Biol. Medicine.

[257]  J. Lowery,et al.  Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science , 2009, Implementation science : IS.

[258]  Julie Morris,et al.  Factors influencing the response to postal questionnaire surveys about respiratory symptoms. , 2008, Primary care respiratory journal : journal of the General Practice Airways Group.

[259]  Jane M. Young,et al.  Audit and feedback: effects on professional practice and healthcare outcomes. , 2012, The Cochrane database of systematic reviews.

[260]  D. Hunter,et al.  A mixed-methods evaluation of transformational change in NHS North East , 2014 .

[261]  Lucy Yardley,et al.  Process evaluation of a point-of-care cluster randomised trial using a computer-delivered intervention to reduce antibiotic prescribing in primary care , 2014, BMC Health Services Research.

[262]  G. Rait,et al.  Strategies to improve retention in randomised trials: a Cochrane systematic review and meta-analysis , 2014, BMJ Open.

[263]  S. Jan,et al.  Vertical equity: weighting outcomes? or establishing procedures? , 1997, Health policy.

[264]  M. Walker,et al.  Towards a programme theory for fidelity in the evaluation of complex interventions. , 2014, Journal of evaluation in clinical practice.

[265]  C. Tuohy Reform and the politics of hybridization in mature health care states. , 2012, Journal of health politics, policy and law.

[266]  M. Clarke,et al.  SWAT 1: what effects do site visits by the principal investigator have on recruitment in a multicentre randomized trial? , 2013, Journal of evidence-based medicine.

[267]  Francisco Nunes,et al.  A longitudinal, multi-level comparative study of quality and safety in European hospitals: the QUASER study protocol , 2011, BMC health services research.

[268]  David M. Cutler,et al.  Equality, Efficiency, and Market Fundamentals: The Dynamics of International Medical Care Reform. , 2002 .

[269]  Stephanie T. Lanza,et al.  Latent Class Analysis: An Alternative Perspective on Subgroup Analysis in Prevention and Treatment , 2013, Prevention Science.

[270]  Karla Hemming,et al.  Multiple component patient safety intervention in English hospitals: controlled evaluation of second phase , 2011, BMJ : British Medical Journal.

[271]  F. Jones,et al.  Lost in translation: exploring therapists’ experiences of providing stroke rehabilitation across a language barrier , 2014, Disability and rehabilitation.

[272]  J T Hart,et al.  THE INVERSE CARE LAW , 1971 .

[273]  Trisha Greenhalgh,et al.  Large-system transformation in health care: a realist review. , 2012, The Milbank quarterly.

[274]  M. Dixon-Woods,et al.  Explaining Matching Michigan: an ethnographic study of a patient safety program , 2013, Implementation Science.

[275]  Susan Leigh Star,et al.  Institutional Ecology, `Translations' and Boundary Objects: Amateurs and Professionals in Berkeley's Museum of Vertebrate Zoology, 1907-39 , 1989 .

[276]  S. Hardoon,et al.  Sociodemographic variations in the contribution of secondary drug prevention to stroke survival at middle and older ages: cohort study , 2009, BMJ : British Medical Journal.

[277]  L. Yardley,et al.  Electronic Health Records for Intervention Research: A Cluster Randomized Trial to Reduce Antibiotic Prescribing in Primary Care (eCRT Study) , 2014, The Annals of Family Medicine.

[278]  K. Hillman,et al.  Introduction of the medical emergency team (MET) system: a cluster-randomised controlled trial , 2005, The Lancet.

[279]  A. Street,et al.  Developing a methodological framework for organisational case studies: a rapid review and consensus development process , 2016 .

[280]  G. Robert,et al.  Diffusion of innovations in service organizations: systematic review and recommendations. , 2004, The Milbank quarterly.

[281]  Hajo Zeeb,et al.  Directed Acyclic Graphs , 2014 .

[282]  C. McKevitt,et al.  Innovations in major system reconfiguration in England: a study of the effectiveness, acceptability and processes of implementation of two models of stroke care , 2013, Implementation Science.

[283]  S. Wyke,et al.  Patient-Reported Outcome Measures for Chronic Obstructive Pulmonary Disease , 2013, The Patient - Patient-Centered Outcomes Research.

[284]  I. Levitas TREATMENT, MODIFICATION AND PREVENTION OF MEASLES BY USE OF IMMUNE GLOBULIN (HUMAN) , 1935 .

[285]  R. Evans,et al.  Public health insurance: the collective purchase of individual care. , 1987, Health policy.

[286]  A. Mayhew,et al.  Interventions to improve professional adherence to guidelines for prevention of device-related infections. , 2013, The Cochrane database of systematic reviews.

[287]  S. Wyke,et al.  Epidemiology of multimorbidity and implications for health care, research, and medical education: a cross-sectional study , 2012, The Lancet.

[288]  N. Black,et al.  Do patient-reported outcomes offer a more sensitive method for comparing the outcomes of consultants than mortality? A multilevel analysis of routine data , 2014, BMJ quality & safety.

[289]  J G Thornton,et al.  Clinical trials and rare diseases: a way out of a conundrum , 1995, BMJ.

[290]  C. Gotay,et al.  The prognostic significance of patient-reported outcomes in cancer clinical trials. , 2008, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.

[291]  P. Bower,et al.  A comprehensive evaluation of the impact of telemonitoring in patients with long-term conditions and social care needs: protocol for the whole systems demonstrator cluster randomised trial , 2011, BMC health services research.

[292]  Celia A Taylor nee Brown,et al.  Systematic review of stepped wedge cluster randomized trials shows that design is particularly used to evaluate interventions during routine implementation. , 2011, Journal of clinical epidemiology.

[293]  G. Lyratzopoulos,et al.  Variation in number of general practitioner consultations before hospital referral for cancer: findings from the 2010 National Cancer Patient Experience Survey in England. , 2012, The Lancet. Oncology.

[294]  R. Pill,et al.  Why do interventions work in some places and not others: a breastfeeding support group trial. , 2010, Social science & medicine.

[295]  Adam Fletcher,et al.  Realist randomised controlled trials: a new approach to evaluating complex public health interventions. , 2012, Social science & medicine.

[296]  Trudie Lang,et al.  Making randomised trials more efficient: report of the first meeting to discuss the Trial Forge platform , 2015, Trials.

[297]  Claire Glenton,et al.  Use of qualitative methods alongside randomised controlled trials of complex healthcare interventions: methodological study , 2009, BMJ : British Medical Journal.

[298]  P. Bower,et al.  Systematic techniques for assisting recruitment to trials (START): study protocol for embedded, randomized controlled trials , 2014, Trials.

[299]  A. Gnanasakthy,et al.  Inclusion of patient-reported outcome measures in registered clinical trials: Evidence from ClinicalTrials.gov (2007-2013). , 2015, Contemporary clinical trials.

[300]  N E Day,et al.  The design of case-control studies: the influence of confounding and interaction effects. , 1984, International journal of epidemiology.

[301]  K. Windle,et al.  Engagement of people with long- term conditions in health and social care research: Barriers and facilitators to capturing the views of seldom-heard populations , 2012 .

[302]  G. Bevan The Search for a Proportionate Care Law by Formula Funding in the English NHS , 2009 .

[303]  M. Orrell,et al.  How does the EQ-5D perform when measuring quality of life in dementia against two other dementia-specific outcome measures? , 2015, Quality of Life Research.

[304]  S. Chinn,et al.  Intraclass correlation coefficient and outcome prevalence are associated in clustered binary data. , 2005, Journal of clinical epidemiology.

[305]  T. Lorenc,et al.  Methods don't make assumptions, researchers do: a response to Marchal et al. , 2013, Social science & medicine.

[306]  A. Bryant,et al.  Patient-reported outcome measures for follow-up after gynaecological cancer treatment. , 2013, The Cochrane database of systematic reviews.

[307]  G. Piaggio,et al.  Consort 2010 statement: extension to cluster randomised trials , 2012, BMJ : British Medical Journal.

[308]  Jeremy M. Grimshaw,et al.  Growing Literature, Stagnant Science? Systematic Review, Meta-Regression and Cumulative Analysis of Audit and Feedback Interventions in Health Care , 2014, Journal of General Internal Medicine.

[309]  R. Raine Bias measuring bias , 2002, Journal of health services research & policy.

[310]  P Barach,et al.  Understanding the complexity of redesigning care around the clinical microsystem , 2006, Quality and Safety in Health Care.

[311]  Allan Donner,et al.  Methods in health service research: Evaluation of health interventions at area and organisation level , 1999 .

[312]  D. Rubin,et al.  The central role of the propensity score in observational studies for causal effects , 1983 .

[313]  R. Glasgow,et al.  Evaluating the public health impact of health promotion interventions: the RE-AIM framework. , 1999, American journal of public health.

[314]  M. Kristenson,et al.  Patient‐reported outcomes in the Swedish National Quality Registers , 2016, Journal of internal medicine.

[315]  S Audrey,et al.  An informal school-based peer-led intervention for smoking prevention in adolescence (ASSIST): a cluster randomised trial , 2008, The Lancet.

[316]  N. Sirven,et al.  Variations in preventive care utilisation in Europe , 2011, European Journal of Ageing.

[317]  Ravi Maheswaran,et al.  Socioeconomic deprivation, travel distance, location of service, and uptake of breast cancer screening in North Derbyshire, UK , 2006, Journal of Epidemiology and Community Health.

[318]  Catherine Sudlow,et al.  Collaborative meta-analysis of randomised trials of antiplatelet therapy for prevention of death, myocardial infarction, and stroke in high risk patients , 2002, BMJ : British Medical Journal.

[319]  A. Hartz,et al.  A comparison of observational studies and randomized, controlled trials , 2000, American journal of ophthalmology.

[320]  A. Leyland,et al.  A multilevel analysis of diet and socio-economic status in Scotland: investigating the ‘Glasgow effect’ , 2009, Public Health Nutrition.

[321]  G. Velikova,et al.  Integrated care pathways for cancer survivors – a role for patient-reported outcome measures and health informatics , 2015, Acta oncologica.

[322]  Robert Chia,et al.  On Organizational Becoming: Rethinking Organizational Change , 2002, Organ. Sci..

[323]  S. Acaster,et al.  Methods for interpreting change over time in patient-reported outcome measures , 2013, Quality of Life Research.

[324]  M. Gulliford,et al.  Cluster randomized trials utilizing primary care electronic health records: methodological issues in design, conduct, and analysis (eCRT Study) , 2014, Trials.

[325]  Margarete Sandelowski,et al.  A Bayesian method for the synthesis of evidence from qualitative and quantitative reports: The example of antiretroviral medication adherence , 2009, Journal of health services research & policy.

[326]  Henna Hasson,et al.  Systematic evaluation of implementation fidelity of complex interventions in health and social care , 2010, Implementation science : IS.

[327]  Kilbourn Jp Letter: Cystic fibrosis. , 1974 .

[328]  P. J. Chilton,et al.  Protocol for evaluation of the cost-effectiveness of ePrescribing systems and candidate prototype for other related health information technologies , 2014, BMC Health Services Research.

[329]  J. Grimshaw,et al.  Changing the behavior of healthcare professionals: the use of theory in promoting the uptake of research findings. , 2005, Journal of clinical epidemiology.

[330]  Andrew M Ryan,et al.  Association of hospital participation in a quality reporting program with surgical outcomes and expenditures for Medicare beneficiaries. , 2015, JAMA.

[331]  A. V. D. Ven,et al.  Process studies of change in organization and management : unveiling temporality, activity, and flow , 2013 .