Evaluation of occupational health interventions using a randomized controlled trial: challenges and alternative research designs.

Occupational health researchers regularly conduct evaluative intervention research for which a randomized controlled trial (RCT) may not be the most appropriate design (eg, effects of policy measures, organizational interventions on work schedules). This article demonstrates the appropriateness of alternative designs for the evaluation of occupational health interventions, which permit causal inferences, formulated along two study design approaches: experimental (stepped-wedge) and observational (propensity scores, instrumental variables, multiple baseline design, interrupted time series, difference-in-difference, and regression discontinuity). For each design, the unique characteristics are presented including the advantages and disadvantages compared to the RCT, illustrated by empirical examples in occupational health. This overview shows that several appropriate alternatives for the RCT design are feasible and available, which may provide sufficiently strong evidence to guide decisions on implementation of interventions in workplaces. Researchers are encouraged to continue exploring these designs and thus contribute to evidence-based occupational health.

[1]  R. C. Hill Regression Discontinuity Designs , 2017 .

[2]  P. Austin Double propensity-score adjustment: A solution to design bias or bias due to incomplete matching , 2016, Statistical methods in medical research.

[3]  Peter Groves,et al.  The 'big data' revolution in healthcare: Accelerating value and innovation , 2016 .

[4]  Stephen J Mooney,et al.  Commentary: Epidemiology in the Era of Big Data , 2015, Epidemiology.

[5]  A. Pronk,et al.  Process Evaluation of an Intervention Program to Reduce Occupational Quartz Exposure Among Dutch Construction Workers , 2015, Journal of occupational and environmental medicine.

[6]  M. Jokela,et al.  Job demands and job strain as risk factors for employee wellbeing in elderly care: an instrumental-variables analysis. , 2015, European journal of public health.

[7]  Till Bärnighausen,et al.  Regression discontinuity designs are underutilized in medicine, epidemiology, and public health: a review of current and best practice. , 2015, Journal of clinical epidemiology.

[8]  Gianluca Baio,et al.  Regression discontinuity designs: an approach to the evaluation of treatment efficacy in primary care using observational data , 2014, BMJ : British Medical Journal.

[9]  H. Hasson,et al.  Functions of behavior change interventions when implementing multi-professional teamwork at an emergency department: a comparative case study , 2014, BMC Health Services Research.

[10]  Natasha E. Noble,et al.  Evaluation of systems-oriented public health interventions: alternative research designs. , 2014, Annual review of public health.

[11]  K. Nielsen,et al.  Opening the black box: Presenting a model for evaluating organizational-level interventions , 2013 .

[12]  Matthew C. Camden,et al.  Absenteeism in Health Care: Using Interlocking Behavioral Contingency Feedback to Increase Attendance With Certified Nursing Assistants , 2013 .

[13]  Dennis Ross-Degnan,et al.  Interrupted time-series analysis yielded an effect estimate concordant with the cluster-randomized controlled trial result. , 2013, Journal of clinical epidemiology.

[14]  W. Van den Noortgate,et al.  Modeling external events in the three-level analysis of multiple-baseline across-participants designs: A simulation study , 2013, Behavior research methods.

[15]  G. Costa,et al.  Are regulations effective in reducing construction injuries? An analysis of the Italian context , 2013, Occupational and Environmental Medicine.

[16]  E. C. Harris,et al.  Evaluation of a case management service to reduce sickness absence. , 2013, Occupational medicine.

[17]  Adam Fletcher,et al.  Realist randomised controlled trials: a new approach to evaluating complex public health interventions. , 2012, Social science & medicine.

[18]  Rik Crutzen,et al.  Researchers should convince policy makers to perform a classic cluster randomized controlled trial instead of a stepped wedge design when an intervention is rolled out. , 2012, Journal of clinical epidemiology.

[19]  Rik Crutzen,et al.  Use of the stepped wedge design cannot be recommended: a critical appraisal and comparison with the classic cluster randomized controlled trial design. , 2012, Journal of clinical epidemiology.

[20]  A. J. van der Beek,et al.  Effectiveness of an intervention at construction worksites on work engagement, social support, physical workload, and need for recovery: results from a cluster randomized controlled trial , 2012, BMC Public Health.

[21]  Stefanie Behncke,et al.  Does retirement trigger ill health? , 2012, Health economics.

[22]  Henk F. van der Molen,et al.  Meeting the Challenges of Implementing an Intervention to Promote Work Ability and Health-Related Quality of Life at Construction Worksites: A Process Evaluation , 2011, Journal of occupational and environmental medicine.

[23]  G. Franklin,et al.  Improving Quality, Preventing Disability and Reducing Costs in Workers' Compensation Healthcare: A Population-based Intervention Study , 2011, Medical care.

[24]  Rebecca R Andridge,et al.  Studies with staggered starts: multiple baseline designs and group-randomized trials. , 2011, American journal of public health.

[25]  Celia A Taylor nee Brown,et al.  Systematic review of stepped wedge cluster randomized trials shows that design is particularly used to evaluate interventions during routine implementation. , 2011, Journal of clinical epidemiology.

[26]  E. Hade,et al.  Cutoff designs for community‐based intervention studies , 2011, Statistics in medicine.

[27]  P. Austin An Introduction to Propensity Score Methods for Reducing the Effects of Confounding in Observational Studies , 2011, Multivariate behavioral research.

[28]  William R. Shadish,et al.  On the Importance of Reliable Covariate Measurement in Selection Bias Adjustments Using Propensity Scores , 2011 .

[29]  D. Bennett An Introduction to Instrumental Variables Analysis: Part 1 , 2010, Neuroepidemiology.

[30]  Peter M. Steiner,et al.  The importance of covariate selection in controlling for selection bias in observational studies. , 2010, Psychological methods.

[31]  Tom Cox,et al.  The future of organizational interventions: Addressing the challenges of today's organizations , 2010 .

[32]  Marco Caliendo,et al.  Benefit Duration, Unemployment Duration and Job Match Quality: A Regression-Discontinuity Approach , 2009, SSRN Electronic Journal.

[33]  M Alan Brookhart,et al.  Instrumental variables I: instrumental variables exploit natural variation in nonexperimental data to estimate causal relationships. , 2009, Journal of clinical epidemiology.

[34]  J Thomas,et al.  Alternatives to randomisation in the evaluation of public-health interventions: statistical analysis and causal inference , 2009, Journal of Epidemiology & Community Health.

[35]  A. Sheikh,et al.  Can We Systematically Review Studies That Evaluate Complex Interventions? , 2009, PLoS medicine.

[36]  E. Faerstein,et al.  A DICTIONARY OF EPIDEMIOLOGY , 2016 .

[37]  D. Heederik,et al.  Effect of an intervention aimed at reducing the risk of allergic respiratory disease in bakers: change in flour dust and fungal alpha-amylase levels , 2009, Occupational and Environmental Medicine.

[38]  Fang Zhang,et al.  Methods for estimating confidence intervals in interrupted time series analyses of health interventions. , 2009, Journal of clinical epidemiology.

[39]  J. Schafer,et al.  Average causal effects from nonrandomized studies: a practical guide and simulated example. , 2008, Psychological methods.

[40]  D. Moher,et al.  Improving the reporting of pragmatic trials: an extension of the CONSORT statement , 2008, BMJ : British Medical Journal.

[41]  Norito Kawakami,et al.  Effects of a Worker Participatory Program for Improving Work Environments on Job Stressors and Mental Health among Workers: A Controlled Trial , 2008, Journal of occupational health.

[42]  M. Petticrew,et al.  Developing and evaluating complex interventions: the new Medical Research Council guidance , 2008, BMJ : British Medical Journal.

[43]  David R. Holtgrave,et al.  Alternatives to the randomized controlled trial. , 2008, American journal of public health.

[44]  Miquel Porta,et al.  A Dictionary of Epidemiology , 2008 .

[45]  P. Hawe,et al.  In response to Spillane V., Byrne M.C., Byrne M., Leathem C.S., O'Malley M. & Cupples M.E. (2007) Monitoring treatment fidelity in a randomized trial of a complex intervention. Journal of Advanced Nursing 60(3), 343-352. Important considerations for standardizing complex interventions. , 2008, Journal of advanced nursing.

[46]  Anthony Shakeshaft,et al.  The multiple baseline design for evaluating population-based research. , 2007, American journal of preventive medicine.

[47]  A. Farmer,et al.  Designing and evaluating complex interventions to improve health care , 2007, BMJ : British Medical Journal.

[48]  J. Hughes,et al.  Design and analysis of stepped wedge cluster randomized trials. , 2007, Contemporary clinical trials.

[49]  R. Lilford,et al.  Bmc Medical Research Methodology Open Access the Stepped Wedge Trial Design: a Systematic Review , 2022 .

[50]  B. Blackwood,et al.  Methodological issues in evaluating complex healthcare interventions. , 2006, Journal of advanced nursing.

[51]  Wiebe R. Pestman,et al.  Instrumental Variables: Application and Limitations , 2006, Epidemiology.

[52]  D. Torgerson,et al.  Cluster randomized controlled trials. , 2005, Journal of evaluation in clinical practice.

[53]  T. Kristensen Intervention studies in occupational epidemiology , 2005, Occupational and Environmental Medicine.

[54]  G. Piaggio,et al.  Consort 2010 statement: extension to cluster randomised trials , 2012, BMJ : British Medical Journal.

[55]  D. Altman,et al.  CONSORT statement: extension to cluster randomised trials , 2004, BMJ : British Medical Journal.

[56]  Lisa M Klesges,et al.  The future of health behavior change research: What is needed to improve translation of research into health promotion practice? , 2004, Annals of behavioral medicine : a publication of the Society of Behavioral Medicine.

[57]  Craig R Ramsay,et al.  INTERRUPTED TIME SERIES DESIGNS IN HEALTH TECHNOLOGY ASSESSMENT: LESSONS FROM TWO SYSTEMATIC REVIEWS OF BEHAVIOR CHANGE STRATEGIES , 2003, International Journal of Technology Assessment in Health Care.

[58]  R. Glasgow,et al.  Why don't we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. , 2003, American journal of public health.

[59]  L. Matowe,et al.  Interrupted Time Series Analysis in Clinical Research , 2003, The Annals of pharmacotherapy.

[60]  Monica Costa Dias,et al.  Alternative approaches to evaluation in empirical microeconomics , 2002, The Journal of Human Resources.

[61]  E. Duflo,et al.  How Much Should We Trust Differences-in-Differences Estimates? , 2001 .

[62]  J. Ioannidis,et al.  Comparison of evidence of treatment effects in randomized and nonrandomized studies. , 2001, JAMA.

[63]  J. Concato,et al.  Randomized, controlled trials, observational studies, and the hierarchy of research designs. , 2000, The New England journal of medicine.

[64]  N. Wolff,et al.  Using randomized controlled trials to evaluate socially complex services: problems, challenges and recommendations. , 2000, The journal of mental health policy and economics.

[65]  Alexander C. Wagenaar,et al.  The Value of Interrupted Time-Series Experiments for Community Intervention Research , 2000, Prevention Science.

[66]  A. Griffiths Organizational interventions: facing the limits of the natural science paradigm. , 1999, Scandinavian journal of work, environment & health.

[67]  R. Glasgow,et al.  Evaluating the public health impact of health promotion interventions: the RE-AIM framework. , 1999, American journal of public health.

[68]  R. D'Agostino Adjustment Methods: Propensity Score Methods for Bias Reduction in the Comparison of a Treatment to a Non‐Randomized Control Group , 2005 .

[69]  J. Angrist,et al.  Empirical Strategies in Labor Economics , 1998 .

[70]  D. Rubin,et al.  The central role of the propensity score in observational studies for causal effects , 1983 .

[71]  A. Cuvo Multiple-baseline design in instructional research: pitfalls of measurement and procedural advantages. , 1979, American journal of mental deficiency.

[72]  D. Gerritsen,et al.  Coming to grips with challenging behaviour: a cluster randomised controlled trial on the effects of a new care programme for challenging behaviour on burnout, job satisfaction and job demands of care staff on dementia special care units. , 2015, International journal of nursing studies.

[73]  张静,et al.  Banana Ovate family protein MaOFP1 and MADS-box protein MuMADS1 antagonistically regulated banana fruit ripening , 2015 .

[74]  Till Bärnighausen,et al.  Regression Discontinuity Designs in Epidemiology: Causal Inference Without Randomized Trials , 2014 .

[75]  M. Kivimäki,et al.  Does perceived work ability improve after a multidisciplinary preventive program in a population with no severe medical problems? The Finnish Public Sector Study. , 2013, Scandinavian journal of work, environment & health.

[76]  M. Kivimäki,et al.  Effectiveness of multidisciplinary primary prevention in decreasing the risk of work disability in a low-risk population. , 2012, Scandinavian journal of work, environment & health.

[77]  Ritva Ketola,et al.  Effectiveness of an ergonomic intervention on the productivity of workers with upper-extremity disorders--a randomized controlled trial. , 2010, Scandinavian journal of work, environment & health.

[78]  William R. Shadish,et al.  Propensity Scores and Quasi-Experiments: A Testimony to the Practical Side of Lee Sechrest. , 2006 .

[79]  I. Steenhuis,et al.  Process evaluation of two environmental nutrition programmes and an educational nutrition programme conducted at supermarkets and worksite cafeterias in the Netherlands. , 2004, Journal of human nutrition and dietetics : the official journal of the British Dietetic Association.

[80]  Francine D. Blau,et al.  Handbook of Labor Economics , 1999 .

[81]  J. Newhouse,et al.  Econometrics in outcomes research: the use of instrumental variables. , 1998, Annual review of public health.