Using Observational Data to Calibrate Simulation Models

Background. Individual-level simulation models are valuable tools for comparing the impact of clinical or public health interventions on population health and cost outcomes over time. However, a key challenge is ensuring that outcome estimates correctly reflect real-world impacts. Calibration to targets obtained from randomized trials may be insufficient if trials do not exist for populations, time periods, or interventions of interest. Observational data can provide a wider range of calibration targets but requires methods to adjust for treatment-confounder feedback. We propose the use of the parametric g-formula to estimate calibration targets and present a case-study to demonstrate its application. Methods. We used the parametric g-formula applied to data from the HIV-CAUSAL Collaboration to estimate calibration targets for 7-y risks of AIDS and/or death (AIDS/death), as defined by the Center for Disease Control and Prevention under 3 treatment initiation strategies. We compared these targets to projections from the Cost-effectiveness of Preventing AIDS Complications (CEPAC) model for treatment-naïve individuals presenting to care in the following year ranges: 1996 to 1999, 2000 to 2002, or 2003 onwards. Results. The parametric g-formula estimated a decreased risk of AIDS/death over time and with earlier treatment. The uncalibrated CEPAC model successfully reproduced targets obtained via the g-formula for baseline 1996 to 1999, but over-estimated calibration targets in contemporary populations and failed to reproduce time trends in AIDS/death risk. Calibration to g-formula targets improved CEPAC model fit for contemporary populations. Conclusion. Individual-level simulation models are developed based on best available information about disease processes in one or more populations of interest, but these processes can change over time or between populations. The parametric g-formula provides a method for using observational data to obtain valid calibration targets and enables updating of simulation model inputs when randomized trials are not available.

[1]  M. Hernán,et al.  Impact of antiretroviral therapy on tuberculosis incidence among HIV-positive patients in high-income countries. , 2012, Clinical infectious diseases : an official publication of the Infectious Diseases Society of America.

[2]  J. Caro,et al.  Model transparency and validation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force--7. , 2012, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[3]  Karen M Kuntz,et al.  Multiparameter calibration of a natural history model of cervical cancer. , 2007, American journal of epidemiology.

[4]  J. Robins,et al.  Intervening on risk factors for coronary heart disease: an application of the parametric g-formula. , 2009, International journal of epidemiology.

[5]  M. Weinstein,et al.  Cost-effectiveness of HIV treatment as prevention in serodiscordant couples. , 2013, The New England journal of medicine.

[6]  M. T. Paixão,et al.  Expanded European AIDS case definition , 1993, The Lancet.

[7]  Jessica G. Young,et al.  Comparative effectiveness of immediate antiretroviral therapy versus CD4-based initiation in HIV-positive individuals in high-income countries: observational cohort study. , 2015, The lancet. HIV.

[8]  Jennifer F Hoy,et al.  Antiretroviral treatment of adult HIV infection: 2014 recommendations of the International Antiviral Society-USA Panel. , 2014, JAMA.

[9]  James M Robins,et al.  The effect of combined antiretroviral therapy on the overall mortality of HIV-infected individuals , 2010, AIDS.

[10]  M. Hughes,et al.  Assessing agreement in the timing of treatment initiation determined by repeated measurements of novel versus gold standard technologies with application to the monitoring of CD4 counts in HIV‐infected patients , 2010, Statistics in medicine.

[11]  James M Robins,et al.  A Comparison of Agent-Based Models and the Parametric G-Formula for Causal Inference , 2017, American journal of epidemiology.

[12]  Paul Windrum,et al.  Empirical Validation of Agent-Based Models: Alternatives and Prospects , 2007, J. Artif. Soc. Soc. Simul..

[13]  A. Saltelli,et al.  Sensitivity Anaysis as an Ingredient of Modeling , 2000 .

[14]  M. Hernán,et al.  Beyond the intention-to-treat in comparative effectiveness research , 2012, Clinical trials.

[15]  J. Savarino,et al.  Bayesian Calibration of Microsimulation Models , 2009, Journal of the American Statistical Association.

[16]  R. Walensky,et al.  Test and treat DC: forecasting the impact of a comprehensive HIV strategy in Washington DC. , 2010, Clinical infectious diseases : an official publication of the Infectious Diseases Society of America.

[17]  Douglas D. Richman,et al.  Antiretroviral Treatment of Adult HIV Infection , 2008 .

[18]  David R Bangsberg,et al.  Predictive value of plasma HIV RNA level on rate of CD4 T-cell decline in untreated HIV infection. , 2006, JAMA.

[19]  Milton C Weinstein,et al.  Empirically calibrated model of hepatitis C virus infection in the United States. , 2002, American journal of epidemiology.

[20]  M C Weinstein,et al.  The cost effectiveness of combination antiretroviral therapy for HIV disease. , 2001, The New England journal of medicine.

[21]  Jakob Grazzini,et al.  Consistent Estimation of Agent Based Models , 2011 .

[22]  Stefano Tarantola,et al.  Sensitivity Analysis as an Ingredient of Modeling , 2000 .

[23]  D. Klein,et al.  Narrowing the Gap in Life Expectancy Between HIV-Infected and HIV-Uninfected Individuals With Access to Care , 2016, Journal of acquired immune deficiency syndromes.

[24]  M. Weinstein,et al.  Assessing the Performance of a Computer-Based Policy Model of HIV and AIDS , 2010, PloS one.

[25]  M C Weinstein,et al.  The cost-effectiveness of preventing AIDS-related opportunistic infections. , 1998, JAMA.

[26]  James M. Robins,et al.  Comparative Effectiveness of Dynamic Treatment Regimes: An Application of the Parametric G-Formula , 2011, Statistics in biosciences.

[27]  Haitao Chu,et al.  The per-protocol effect of immediate versus deferred antiretroviral therapy initiation , 2016, AIDS.

[28]  K. White,et al.  Co-formulated elvitegravir, cobicistat, emtricitabine, and tenofovir versus co-formulated efavirenz, emtricitabine, and tenofovir for initial treatment of HIV-1 infection: a randomised, double-blind, phase 3 trial, analysis of results after 48 weeks , 2012, The Lancet.

[29]  J. Sterne,et al.  Survival of HIV-positive patients starting antiretroviral therapy between 1996 and 2013: a collaborative analysis of cohort studies , 2017, The lancet. HIV.

[30]  Sandro Galea,et al.  Formalizing the role of agent-based modeling in causal inference and epidemiology. , 2015, American journal of epidemiology.

[31]  John T Brooks,et al.  Guidelines for prevention and treatment of opportunistic infections in HIV-infected adults and adolescents: recommendations from CDC, the National Institutes of Health, and the HIV Medicine Association of the Infectious Diseases Society of America. , 2009, MMWR. Recommendations and reports : Morbidity and mortality weekly report. Recommendations and reports.

[32]  Ann G Zauber,et al.  National Polyp Study data: Evidence for regression of adenomas , 2004, International journal of cancer.

[33]  James M Robins,et al.  Smoking cessation and long-term weight gain in the Framingham Heart Study: an application of the parametric g-formula for a continuous outcome , 2016, European Journal of Epidemiology.

[34]  Marc Lipsitch,et al.  Development, Calibration and Performance of an HIV Transmission Model Incorporating Natural History and Behavioral Patterns: Application in South Africa , 2014, PloS one.

[35]  Stephen R Cole,et al.  Timing of initiation of antiretroviral therapy in AIDS-free HIV-1-infected patients: a collaborative analysis of 18 HIV cohort studies , 2009, The Lancet.

[36]  M. Robins James,et al.  Estimation of the causal effects of time-varying exposures , 2008 .

[37]  B. Gazzard,et al.  CD4 Cell Count and the Risk of AIDS or Death in HIV-Infected Adults on Combination Antiretroviral Therapy with a Suppressed Viral Load: A Longitudinal Cohort Study from COHERE , 2012, PLoS medicine.

[38]  D. Owens,et al.  State-transition modeling: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force--3. , 2012, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[39]  Amalio Telenti,et al.  Antiretroviral Treatment of Adult HIV Infection2010 Recommendations of the International AIDS Society–USA Panel , 2010 .

[40]  Milton C Weinstein,et al.  Principles of good practice for decision analytic modeling in health-care evaluation: report of the ISPOR Task Force on Good Research Practices--Modeling Studies. , 2003, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[41]  K. Freedberg,et al.  Association Between Medication Possession Ratio, Virologic Failure and Drug Resistance in HIV-1–Infected Adults on Antiretroviral Therapy in Côte d'Ivoire , 2011, Journal of acquired immune deficiency syndromes.

[42]  J. Robins.  A new approach to causal inference in mortality studies with a sustained exposure period—application to control of the healthy worker survivor effect , 1986 .

[43]  James M Robins,et al.  When to Initiate Combined Antiretroviral Therapy to Reduce Mortality and AIDS-Defining Illness in HIV-Infected Persons in Developed Countries , 2011, Annals of Internal Medicine.

[44]  James M Robins,et al.  Randomized Trials Analyzed as Observational Studies , 2013, Annals of Internal Medicine.

[45]  M. Weinstein,et al.  Validation and Calibration of a Computer Simulation Model of Pediatric HIV Infection , 2013, PloS one.