Calibration of individual-based models to epidemiological data: A systematic review

Individual-based models (IBMs) informing public health policy should be calibrated to data and provide estimates of uncertainty. Two main components of model-calibration methods are the parameter-search strategy and the goodness-of-fit (GOF) measure; many options exist for each of these. This review provides an overview of calibration methods used in IBMs modelling infectious disease spread. We identified articles on PubMed employing simulation-based methods to calibrate IBMs informing public health policy in HIV, tuberculosis, and malaria epidemiology published between 1 January 2013 and 31 December 2018. Articles were included if models stored individual-specific information, and calibration involved comparing model output to population-level targets. We extracted information on parameter-search strategies, GOF measures, and model validation. The PubMed search identified 653 candidate articles, of which 84 met the review criteria. Of the included articles, 40 (48%) combined a quantitative GOF measure with an algorithmic parameter-search strategy–either an optimisation algorithm (14/40) or a sampling algorithm (26/40). These 40 articles varied widely in their choices of parameter-search strategies and GOF measures. For the remaining 44 (52%) articles, the parameter-search strategy could either not be identified (32/44) or was described as an informal, non-reproducible method (12/44). Of these 44 articles, the majority (25/44) were unclear about the GOF measure used; of the rest, only five quantitatively evaluated GOF. Only a minority of the included articles, 14 (17%) provided a rationale for their choice of model-calibration method. Model validation was reported in 31 (37%) articles. Reporting on calibration methods is far from optimal in epidemiological modelling studies of HIV, malaria and TB transmission dynamics. The adoption of better documented, algorithmic calibration methods could improve both reproducibility and the quality of inference in model-based epidemiology. There is a need for research comparing the performance of calibration methods to inform decisions about the parameter-search strategies and GOF measures.

[1]  Jonathan Karnon,et al.  Model Performance Evaluation (Validation and Calibration) in Model-based Studies of Therapeutic Interventions for Cardiovascular Diseases , 2013, Applied Health Economics and Health Policy.

[2]  Richard G. White,et al.  Small contribution of gold mines to the ongoing tuberculosis epidemic in South Africa: a modeling-based study , 2018, BMC Medicine.

[3]  Marc Lipsitch,et al.  Development, Calibration and Performance of an HIV Transmission Model Incorporating Natural History and Behavioral Patterns: Application in South Africa , 2014, PloS one.

[4]  Nathan Geffen,et al.  A Comparison of Two Mathematical Modeling Frameworks for Evaluating Sexually Transmitted Infection Epidemiology , 2016, Sexually transmitted diseases.

[5]  Aki Vehtari,et al.  Validating Bayesian Inference Algorithms with Simulation-Based Calibration , 2018, 1804.06788.

[6]  Forrest Stonedahl,et al.  The Complexities of Agent-Based Modeling Output Analysis , 2015, J. Artif. Soc. Soc. Simul..

[7]  D. Rubin.  Using the SIR algorithm to simulate posterior distributions , 1988 .

[8]  Alexander Y. Sun,et al.  Model Calibration and Parameter Estimation: For Environmental and Water Resource Systems , 2015 .

[9]  R. Leombruni,et al.  Why are economists sceptical about agent-based simulations? , 2005 .

[10]  Andreas Huth,et al.  Statistical inference for stochastic simulation models--theory and application. , 2011, Ecology letters.

[11]  Jonathan Karnon,et al.  Model Parameter Estimation and Uncertainty Analysis , 2012, Medical decision making : an international journal of the Society for Medical Decision Making.

[12]  Wim Delva,et al.  Differential sexual network connectivity offers a parsimonious explanation for population-level variations in the prevalence of bacterial vaginosis: a data-driven, model-supported hypothesis , 2019, BMC Women's Health.

[13]  Douglas A. Wolfe,et al.  Nonparametric Statistical Methods , 1973 .

[14]  Niel Hens,et al.  Lessons from a decade of individual-based models for infectious disease transmission: a systematic review (2006-2015) , 2017, BMC Infectious Diseases.

[15]  Susan L Norris,et al.  Developing WHO guidelines: Time to formally include evidence from mathematical modelling studies , 2017, F1000Research.

[16]  Leigh F. Johnson,et al.  MicroCOSM: a model of social and structural drivers of HIV and interventions to reduce HIV incidence in high-risk populations in South Africa , 2018, bioRxiv.

[17]  Jonathan Karnon,et al.  Calibrating Models in Economic Evaluation , 2012, PharmacoEconomics.

[18]  Natasha K. Stout,et al.  Calibration Methods Used in Cancer Simulation Models and Suggested Reporting Guidelines , 2012, PharmacoEconomics.

[19]  A. O'Hagan,et al.  Bayesian calibration of computer models , 2001 .

[20]  Joachim M. Buhmann,et al.  Stable Bayesian Parameter Estimation for Biological Dynamical Systems , 2009, 2009 International Conference on Computational Science and Engineering.

[21]  Anya Okhmatovskaia,et al.  Validation of population-based disease simulation models: a review of concepts and methods , 2010, BMC public health.

[22]  Philip H. Ramsey.  Nonparametric Statistical Methods , 1974, Technometrics.

[23]  Ye Chen,et al.  Implementation and applications of EMOD, an individual-based multi-disease modeling platform , 2018, Pathogens and disease.

[24]  Roel Bakker,et al.  STDSIM: A Microsimulation Model for Decision Support in the Control of HIV and Other STDs , 2000 .

[25]  Robert J. Morris,et al.  Uncertainty and Inference in Agent-Based Models , 2010, 2010 Second International Conference on Advances in System Simulation.

[26]  John A. Nelder,et al.  A Simplex Method for Function Minimization , 1965, Comput. J..

[27]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[28]  Ross A. Hammond.  Considerations and Best Practices in Agent-Based Modeling to Inform Policy , 2015 .

[29]  Klaus Keller,et al.  Small increases in agent-based model complexity can result in large increases in required calibration data , 2018 .

[30]  Philip A. Eckhoff,et al.  Targeting HIV services to male migrant workers in southern Africa would not reverse generalized HIV epidemics in their home communities: a mathematical modeling analysis , 2015, International health.

[31]  Yevgeniy Vorobeychik,et al.  Empirically grounded agent-based models of innovation diffusion: a critical review , 2016, Artificial Intelligence Review.

[32]  D. Moher,et al.  Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement , 2009, BMJ : British Medical Journal.

[33]  Martin Bicher,et al.  A Systematic Review Of Calibration In Population Models , 2017 .

[34]  Richard G. White,et al.  Calibrating Models in Economic Evaluation , 2012, PharmacoEconomics.

[35]  Jeremy E. Oakley,et al.  Universal test, treat, and keep: improving ART retention is key in cost-effective HIV control in Uganda , 2017, BMC Infectious Diseases.

[36]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[37]  Chaitra Gopalappa,et al.  Progression and Transmission of HIV/AIDS (PATH 2.0) , 2017, Medical decision making : an international journal of the Society for Medical Decision Making.

[38]  Philip A. Eckhoff,et al.  Age-dependent partnering and the HIV transmission chain: a microsimulation analysis , 2013, Journal of The Royal Society Interface.

[39]  H. Stryhn,et al.  Confidence intervals by the profile likelihood method , with applications in veterinary epidemiology , 2003 .

[40]  M. Gutmann,et al.  Fundamentals and Recent Developments in Approximate Bayesian Computation , 2016, Systematic biology.

[41]  David J Gerberry,et al.  An exact approach to calibrating infectious disease models to surveillance data: The case of HIV and HSV-2. , 2017, Mathematical biosciences and engineering : MBE.

[42]  Katya Galactionova,et al.  The public health impact of malaria vaccine RTS,S in malaria endemic Africa: country-specific predictions using 18 month follow-up Phase III data and simulation models , 2015, BMC Medicine.

[43]  G. Seage,et al.  Individual-Based Simulation Models of HIV Transmission: Reporting Quality and Recommendations , 2013, PloS one.

[44]  Uwe Siebert,et al.  Modeling good research practices--overview: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force--1. , 2012, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[45]  James M Robins,et al.  Using Observational Data to Calibrate Simulation Models , 2017, Medical decision making : an international journal of the Society for Medical Decision Making.

[46]  Y. Furuse,et al.  Analysis of research intensity on infectious disease by disease burden reveals which infectious diseases are neglected by researchers , 2018, Proceedings of the National Academy of Sciences.

[47]  L. Meyers,et al.  Probabilistic uncertainty analysis of epidemiological modeling to guide public health intervention policy. , 2014, Epidemics.

[48]  Jason Kessler,et al.  Impact and Cost-Effectiveness of Hypothetical Strategies to Enhance Retention in Care within HIV Treatment Programs in East Africa. , 2015, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[49]  Jens Saak,et al.  Best Practices for Replicability, Reproducibility and Reusability of Computer-Based Experiments Exemplified by Model Reduction Software , 2016, ArXiv.

[50]  Susan L Norris,et al.  Developing WHO guidelines: Time to formally include evidence from mathematical modelling studies , 2017, F1000Research.

[51]  Thomas A Louis,et al.  Mathematical Modeling of “Chronic” Infectious Diseases: Unpacking the Black Box , 2017, Open forum infectious diseases.

[52]  Nikolaos V. Sahinidis,et al.  Simulation optimization: a review of algorithms and applications , 2014, 4OR.

[53]  D. Moher,et al.  Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. , 2010, International journal of surgery.

[54]  Jeremy E. Oakley,et al.  Approximate Bayesian Computation and simulation based inference for complex stochastic epidemic models , 2018 .

[55]  A. Raftery,et al.  Inference for Deterministic Simulation Models: The Bayesian Melding Approach , 2000 .

[56]  Amy Earley,et al.  Modeling and Simulation in the Context of Health Technology Assessment: Review of Existing Guidance, Future Research Needs, and Validity Assessment , 2017 .

[57]  C. Schunn,et al.  Evaluating Goodness-of-Fit in Comparison of Models to Data , 2005 .

[58]  Nicolas A. Menzies,et al.  Bayesian Methods for Calibrating Health Policy Models: A Tutorial , 2017, PharmacoEconomics.

[59]  A Kremling,et al.  Exploiting the bootstrap method for quantifying parameter confidence intervals in dynamical systems. , 2006, Metabolic engineering.

[60]  Milton C Weinstein,et al.  The Anticipated Clinical and Economic Effects of 909090 in South Africa , 2016, Annals of Internal Medicine.

[61]  André Grow,et al.  An Agent-Based Model of Status Construction in Task Focused Groups , 2015, J. Artif. Soc. Soc. Simul..

[62]  Daniela De Angelis,et al.  Mathematical models for the study of HIV spread and control amongst men who have sex with men , 2011, European Journal of Epidemiology.

[63]  James S. Hodges,et al.  Six (Or So) Things You Can Do with a Bad Model , 1991, Oper. Res..

[64]  T. Trikalinos,et al.  A Review of Validation and Calibration Methods for Health Care Modeling and Simulation , 2017 .

[65]  Andrew H Briggs,et al.  Questionnaire to assess relevance and credibility of modeling studies for informing health care decision making: an ISPOR-AMCP-NPC Good Practice Task Force report. , 2014, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[66]  Sean R Eddy,et al.  What is dynamic programming? , 2004, Nature Biotechnology.

[67]  Vivek Pawar,et al.  Methods of Model Calibration , 2012, PharmacoEconomics.

[68]  Wim Delva,et al.  Connecting the dots: network data and models in HIV epidemiology , 2016, AIDS.