Evaluating the robustness of targeted maximum likelihood estimators via realistic simulations in nutrition intervention trials

Correspondence *Alan E Hubbard, Division of Biostatistics, University of California, Berkeley, 2121 Berkeley Way Rm 5302, Berkeley, CA 94720-7360 Email: hubbard@berkeley.edu Summary Several recently developed methods have the potential to harness machine learning in the pursuit of target quantities inspired by causal inference, including inverse weighting, doubly robust estimating equations and substitution estimators like targeted maximum likelihood estimation. There are even more recent augmentations of these procedures that can increase robustness, by adding a layer of cross-validation (cross-validated targeted maximum likelihood estimation and double machine learning, as applied to substitution and estimating equation approaches, respectively). While these methods have been evaluated individually on simulated and experimental data sets, a comprehensive analysis of their performance across “real-world” simulations have yet to be conducted. In this work, we benchmark multiple widely used methods for estimation of the average treatment effect using ten different nutrition intervention studies data. A realistic set of simulations, based on a novel method, highly adaptive lasso, for estimating the data-generating distribution that guarantees a certain level of complexity (undersmoothing) is used to better mimic the complexity of the true data-generating distribution. We have applied this novel method for estimating the data-generating distribution by individual study and to subsequently use these fits to simulate data and estimate treatment effects parameters as well as their standard errors and resulting confidence intervals. Based on the analytic results, a general recommendation is put forth for use of the cross-validated variants of both substitution and estimating equation estimators. We conclude that the additional layer of cross-validation helps in avoiding unintentional over-fitting of nuisance parameter functionals and leads to more robust inferences.

[1]  M. J. van der Laan,et al.  Association of Implementation of a Universal Testing and Treatment Intervention With HIV Diagnosis, Receipt of Antiretroviral Therapy, and Viral Suppression in East Africa , 2017, JAMA.

[2]  M. J. van der Laan,et al.  Early childhood linear growth failure in low- and middle-income countries , 2020, medRxiv.

[3]  A. Massie,et al.  Effect of maternal multiple micronutrient vs iron-folic acid supplementation on infant mortality and adverse birth outcomes in rural Bangladesh: The JiVitA-3 randomized trial , 2015 .

[4]  E. Stuart,et al.  Moving towards best practice when using inverse probability of treatment weighting (IPTW) using the propensity score to estimate causal treatment effects in observational studies , 2015, Statistics in medicine.

[5]  Mark van der Laan,et al.  Population Intervention Causal Effects Based on Stochastic Interventions , 2012, Biometrics.

[6]  J. Sekhon,et al.  Evaluating treatment effectiveness under model misspecification: A comparison of targeted maximum likelihood estimation with bias-corrected matching , 2014, Statistical methods in medical research.

[7]  Antoine Chambaz,et al.  Scalable collaborative targeted learning for high-dimensional data , 2017, Statistical methods in medical research.

[8]  Mark J van der Laan,et al.  A practical illustration of the importance of realistic individualized treatment rules in causal inference. , 2007, Electronic journal of statistics.

[9]  K. Maleta,et al.  Prevention and treatment of childhood malnutrition in rural Malawi: Lungwena nutrition studies. , 2009, Malawi medical journal : the journal of Medical Association of Malawi.

[10]  M. J. van der Laan,et al.  Doubly Robust and Efficient Estimation of Marginal Structural Models for the Hazard Function , 2016, The international journal of biostatistics.

[11]  Mark J van der Laan,et al.  An Application of Collaborative Targeted Maximum Likelihood Estimation in Causal Inference and Genomics , 2010, The international journal of biostatistics.

[12]  Mercedes Onis,et al.  WHO Child Growth Standards based on length/height, weight and age , 2006, Acta paediatrica (Oslo, Norway : 1992). Supplement.

[13]  Mark J van der Laan,et al.  An Application of Targeted Maximum Likelihood Estimation to the Meta‐Analysis of Safety Data , 2013, Biometrics.

[14]  M. J. van der Laan,et al.  The International Journal of Biostatistics Collaborative Double Robust Targeted Maximum Likelihood Estimation , 2011 .

[15]  D. Rubin.  [On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9.] Comment: Neyman (1923) and Causal Inference in Experiments and Observational Studies , 1990 .

[16]  M. J. van der Laan,et al.  The International Journal of Biostatistics Targeted Maximum Likelihood Learning , 2011 .

[17]  Andreas Ziegler,et al.  ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R , 2015, 1508.04409.

[18]  J. Robins.  A new approach to causal inference in mortality studies with a sustained exposure period—application to control of the healthy worker survivor effect , 1986 .

[19]  T. Clasen,et al.  Effects of water quality, sanitation, handwashing, and nutritional interventions on diarrhoea and child growth in rural Bangladesh: a cluster randomised controlled trial , 2018, The Lancet. Global health.

[20]  Holly N. Dentz,et al.  Effects of water quality, sanitation, handwashing, and nutritional interventions on child development in rural Kenya (WASH Benefits Kenya): a cluster-randomised controlled trial , 2018, The Lancet. Child & adolescent health.

[21]  Mark J. van der Laan,et al.  Asymptotic Theory for Cross-validated Targeted Maximum Likelihood Estimation , 2010 .

[22]  Denis Talbot,et al.  A generalized double robust Bayesian model averaging approach to causal effect estimation with application to the study of osteoporotic fractures , 2020, Journal of Causal Inference.

[23]  James M. Robins,et al.  Unified Methods for Censored Longitudinal Data and Causality , 2003 .

[24]  S. de Pee,et al.  Effect of fortified complementary food supplementation on child growth in rural Bangladesh: a cluster-randomized trial , 2015, International journal of epidemiology.

[25]  Mark J. van der Laan,et al.  The Highly Adaptive Lasso Estimator , 2016, 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA).

[26]  Trevor Hastie,et al.  Regularization Paths for Generalized Linear Models via Coordinate Descent. , 2010, Journal of statistical software.

[27]  M. J. Laan,et al.  Targeted Learning: Causal Inference for Observational and Experimental Data , 2011 .

[28]  Jeremy Robert Coyle,et al.  Computational Considerations for Targeted Learning , 2017 .

[29]  Oleg Sofrygin,et al.  Complier Stochastic Direct Effects: Identification and Robust Estimation , 2018, Journal of the American Statistical Association.

[30]  J. Robins,et al.  Double/Debiased Machine Learning for Treatment and Structural Parameters , 2017 .

[31]  Tianqi Chen,et al.  XGBoost: A Scalable Tree Boosting System , 2016, KDD.

[32]  M. J. Laan,et al.  Construction of Counterfactuals and the G-computation Formula , 2002 .

[33]  O. Arah,et al.  G-computation of average treatment effects on the treated and the untreated , 2017, BMC Medical Research Methodology.

[34]  D. Horvitz,et al.  A Generalization of Sampling Without Replacement from a Finite Universe , 1952 .

[35]  Adam Glynn,et al.  An Introduction to the Augmented Inverse Propensity Weighted Estimator , 2010, Political Analysis.

[36]  Iván Díaz,et al.  Targeted Data Adaptive Estimation of the Causal Dose–Response Curve , 2013 .

[37]  J. Wellner,et al.  Inefficient estimators of the bivariate survival function for three models , 1995 .

[38]  Sherri Rose,et al.  Robust Machine Learning Variable Importance Analyses of Medical Conditions for Health Care Spending , 2018, Health services research.

[39]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[40]  Nima S. Hejazi,et al.  hal9001: Scalable highly adaptive lasso regression in R , 2020, J. Open Source Softw..

[41]  Mark J van der Laan,et al.  EFFECT OF BREASTFEEDING ON GASTROINTESTINAL INFECTION IN INFANTS: A TARGETED MAXIMUM LIKELIHOOD APPROACH FOR CLUSTERED LONGITUDINAL DATA. , 2014, The annals of applied statistics.

[42]  M. J. van der Laan,et al.  Child wasting and concurrent stunting in low- and middle-income countries , 2020, Nature.

[43]  B. Birren,et al.  Genetic Diversity and Protective Efficacy of the RTS , S / AS 01 Malaria Vaccine , 2015 .

[44]  M. Bhan,et al.  Food supplementation with encouragement to feed it to infants from 4 to 12 months of age has a small impact on weight gain. , 2001, The Journal of nutrition.

[45]  Kristin E. Porter,et al.  Diagnosing and responding to violations in the positivity assumption , 2012, Statistical methods in medical research.

[46]  R. Tibshirani,et al.  Generalized Additive Models , 1986 .

[47]  Bianca L. De Stavola,et al.  Gformula: Estimating Causal Effects in the Presence of Time-Varying Confounding or Mediation using the G-Computation Formula , 2011 .

[48]  J. Skeem,et al.  Comparing Public Safety Outcomes for Traditional Probation vs Specialty Mental Health Probation , 2017, JAMA psychiatry.

[49]  E. Moodie,et al.  Targeted maximum likelihood estimation for marginal time-dependent treatment effects under density misspecification. , 2013, Biostatistics.

[50]  Susan Gruber,et al.  Variable Selection for Confounder Control, Flexible Modeling and Collaborative Targeted Minimum Loss-Based Estimation in Causal Inference , 2016, The international journal of biostatistics.

[51]  M. J. van der Laan,et al.  Simple Optimal Weighting of Cases and Controls in Case-Control Studies , 2008, The international journal of biostatistics.

[52]  Mireille E Schnitzer,et al.  Double robust and efficient estimation of a prognostic model for events in the presence of dependent censoring. , 2015, Biostatistics.

[53]  Azza Abouzeid,et al.  Is this Real?: Generating Synthetic Data that Looks Real , 2019, UIST.

[54]  Alexander Breskin,et al.  Machine Learning for Causal Inference: On the Use of Cross-fit Estimators , 2021, Epidemiology.

[55]  Mireille E Schnitzer,et al.  Understanding and diagnosing the potential for bias when using machine learning methods with doubly robust causal estimators , 2019, Statistical methods in medical research.

[56]  M. Davidian,et al.  Covariate adjustment for two‐sample treatment comparisons in randomized clinical trials: A principled yet flexible approach , 2008, Statistics in medicine.

[57]  B. Giraudeau,et al.  G-computation, propensity score-based methods, and targeted maximum likelihood estimator for causal inference with different covariates sets: a comparative simulation study , 2020, Scientific Reports.

[58]  Cheng Ju,et al.  Collaborative-controlled LASSO for constructing propensity score-based estimators in high-dimensional data , 2017, Statistical methods in medical research.

[59]  Y. Cheung,et al.  Supplementation of Maternal Diets during Pregnancy and for 6 Months Postpartum and Infant Diets Thereafter with Small-Quantity Lipid-Based Nutrient Supplements Does Not Promote Child Growth by 18 Months of Age in Rural Malawi: A Randomized Controlled Trial. , 2015, The Journal of nutrition.

[60]  M. J. van der Laan,et al.  A fundamental measure of treatment effect heterogeneity , 2018, Journal of Causal Inference.

[61]  Benjamin F. Arnold,et al.  Causes and consequences of child growth failure in low- and middle-income countries , 2020, medRxiv.

[62]  Y. Cheung,et al.  Provision of 10-40 g/d Lipid-Based Nutrient Supplements from 6 to 18 Months of Age Does Not Prevent Linear Growth Faltering in Malawi. , 2015, The Journal of nutrition.

[63]  M. J. van der Laan,et al.  On adaptive propensity score truncation in causal inference , 2017, Statistical Methods in Medical Research.

[64]  James M. Robins,et al.  Marginal Structural Models versus Structural nested Models as Tools for Causal inference , 2000 .

[65]  M. J. van der Laan,et al.  Targeted Minimum Loss Based Estimation of Causal Effects of Multiple Time Point Interventions , 2012, The international journal of biostatistics.

[66]  Miguel Angel Luque-Fernandez,et al.  Targeted maximum likelihood estimation for a binary treatment: A tutorial , 2018, Statistics in medicine.

[67]  T. Schuster,et al.  Effect Estimation in Point-Exposure Studies with Binary Outcomes and High-Dimensional Covariate Data – A Comparison of Targeted Maximum Likelihood Estimation and Inverse Probability of Treatment Weighting , 2016, The international journal of biostatistics.

[68]  Mark J van der Laan,et al.  The International Journal of Biostatistics Collaborative Targeted Maximum Likelihood for Time to Event Data , 2011 .

[69]  A Data-Adaptive Targeted Learning Approach of Evaluating Viscoelastic Assay Driven Trauma Treatment Protocols , 2019, 1909.12881.

[70]  D. Rubin,et al.  The central role of the propensity score in observational studies for causal effects , 1983 .

[71]  M J van der Laan,et al.  Covariate adjustment in randomized trials with binary outcomes: Targeted maximum likelihood estimation , 2009, Statistics in medicine.

[72]  Peter M. Aronow,et al.  Estimating Average Causal Effects Under Interference Between Units , 2013, 1305.6156.

[73]  A. Kaufman,et al.  Targeted Estimation of the Relationship Between Childhood Adversity and Fluid Intelligence in a US Population Sample of Adolescents , 2018, American journal of epidemiology.

[74]  Bianca L. De Stavola,et al.  Gformula: Estimating Causal Effects in the Presence of Time-Varying Confounding or Mediation using the G-Computation Formula , 2011 .

[75]  M. J. van der Laan,et al.  Targeted maximum likelihood estimation in safety analysis. , 2013, Journal of clinical epidemiology.

[76]  Sherri Rose,et al.  Targeted Maximum Likelihood Estimation for Causal Inference in Observational Studies , 2017, American journal of epidemiology.

[77]  J. Ouédraogo,et al.  Small-Quantity Lipid-Based Nutrient Supplements, Regardless of Their Zinc Content, Increase Growth and Reduce the Prevalence of Stunting and Wasting in Young Burkinabe Children: A Cluster-Randomized Trial , 2015, PloS one.