Modeling and Simulation in the Context of Health Technology Assessment: Review of Existing Guidance, Future Research Needs, and Validity Assessment

Background Despite rigorous systematic reviews of efficacy and effectiveness of health care interventions, patients, providers and policymakers may remain in doubt about what they should do because of uncertainty, tradeoffs among benefits and harms, and conflicting preferences. Modeling and simulation studies in health care can supplement systematic reviews to increase the usefulness of the evidence summary. The aims of this report are four-fold: (1) to summarize evidence- and consensus-based guidance on the conduct and reporting of health care modeling and simulation; (2) to summarize guidance from Health Technology Assessment (HTA) groups for modeling; (3) to prioritize future research needs to improve models; and (4) to provide an overview of methods for model calibration and validation. Methods With guidance from a Technical Expert Panel and a Clinical and Policy Advisory team with clinical, methodological research, and policymaking expertise, we completed the following projects: For Aim 1, we conducted a systematic review of articles that provided evidence- or consensus-based recommendations for the conduct and reporting of health care modeling and simulation studies. We classified recommendation statements into four domains: model structure, data, consistency, and communication of model results. To contextualize the findings of the systematic review, we organized a meeting with a group of 28 stakeholders, including modelers, users of models, and funders of research. For Aim 2, we searched the web sites of 126 international agencies and institutes conducting HTA for real-world practices regarding when to apply modeling and simulation methods, which we summarized. For Aim 3, from the systematic review and from the stakeholders in Aim 1 we identified and collected suggestions for future research needs. Stakeholders prioritized those needs based on importance, desirability of new research, feasibility, and potential impact. For Aim 4, we searched for articles that compared or applied alternative validation methods for modeling and simulation. We extracted and summarized descriptions and comparisons of any methods and reported results for face validity, internal validity, external validity, cross-model validation, and calibration. Results The systematic review of modeling recommendations (Aim 1) found 71 eligible articles. 90 percent of articles (n=64) provided recommendations regarding model structure. Almost all articles (n=68, 96%) also provided recommendations regarding obtaining appropriate data to populate models. Stakeholders highlighted the importance of establishing guiding principles for “good practice” but discouraged the use of “cookbook” checklists. Of the 71 articles, 38 (54%) provided suggestions for future research; stakeholders provided 28 additional suggestions. We found 21 HTA organizations provided guidance (Aim 2) through their web sites regarding the application of modeling and simulation in the context of conducting a HTA. The HTA organizations varied widely in what areas of modeling they provided guidance for and what specific recommendations they made. In general, HTA organizations favored incorporating models into HTA, provided recommendations on how to model data and structure, and recommended inclusion of costs in cost-effectiveness models. Future research needs that were prioritized (Aim 3) included questions about model data, model structure, consistency, and reporting. Studies comparing validation methods (Aim 4) provided information on model validation (face validity and internal, external, and cross-model validation) and calibration (varying specifications of the calibration problem with the same and different algorithms and use of alternative algorithms for the same calibration problem). Conclusion Our systematic review and stakeholder meeting summary provides a comprehensive compendium of guidance documents for modeling and simulation studies, annotated with information on the domains covered by each document, and the methods used to arrive at specific recommendations. We also summarized modeling recommendations for HTA organizations. These processes enabled us to prioritize future research needs to form an empirical basis for and to improve recommendations for modeling. Our overview of model validation and calibration provides insights into the relative value and efficiency of different methods.