Model Transparency and Validation

Trust and confidence are critical to the success of health care models. There are two main methods for achieving this: transparency (people can see how the model is built) and validation (how well it reproduces reality). This report describes recommendations for achieving transparency and validation, developed by a task force appointed by the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) and the Society for Medical Decision Making (SMDM). Recommendations were developed iteratively by the authors. A nontechnical description should be made available to anyone—including model type and intended applications; funding sources; structure; inputs, outputs, other components that determine function, and their relationships; data sources; validation methods and results; and limitations. Technical documentation, written in sufficient detail to enable a reader with necessary expertise to evaluate the model and potentially reproduce it, should be made available openly or under agreements that protect intellectual property, at the discretion of the modelers. Validation involves face validity (wherein experts evaluate model structure, data sources, assumptions, and results), verification or internal validity (check accuracy of coding), cross validity (comparison of results with other models analyzing same problem), external validity (comparing model results to real-world results), and predictive validity (comparing model results with prospectively observed events). The last two are the strongest form of validation. Each section of this paper contains a number of recommendations that were iterated among the authors, as well as the wider modeling task force jointly set up by the International Society for Pharmacoeconomics and Outcomes Research and the Society for Medical Decision Making.

[1]  Ron Goeree,et al.  Principles of good modeling practice in healthcare cost-effectiveness studies , 2004, Expert review of pharmacoeconomics & outcomes research.

[2]  P. Royle,et al.  Literature searching for clinical and cost-effectiveness studies used in health technology assessment reports carried out for the National Institute for Clinical Excellence appraisal system. , 2003, Health technology assessment.

[3]  M C Weinstein,et al.  The cost effectiveness of combination antiretroviral therapy for HIV disease. , 2001, The New England journal of medicine.

[4]  M. Caulfield,et al.  Effects of torcetrapib in patients at high risk for coronary events. , 2007, The New England journal of medicine.

[5]  Milton C Weinstein,et al.  Expanded screening for HIV in the United States--an analysis of cost-effectiveness. , 2005, The New England journal of medicine.

[6]  K. A. McKibbon,et al.  Optimal search strategies for retrieving scientifically strong studies of treatment from Medline: analytical survey , 2005, BMJ : British Medical Journal.

[7]  L. Goldman,et al.  Forecasting coronary heart disease incidence, mortality, and cost: the Coronary Heart Disease Policy Model. , 1987, American journal of public health.

[8]  Ken Williams,et al.  Validation of Prediction of Diabetes by the Archimedes Model and Comparison With Other Predicting Models , 2008, Diabetes Care.

[9]  D. Berry,et al.  Effect of screening and adjuvant therapy on mortality from breast cancer. , 2006, The New England journal of medicine.

[10]  David M Eddy,et al.  Validation of the archimedes diabetes model. , 2003, Diabetes care.

[11]  Milton C Weinstein,et al.  Cost-effectiveness of HIV treatment in resource-poor settings--the case of Côte d'Ivoire. , 2006, The New England journal of medicine.

[12]  Paul Bratley,et al.  A guide to simulation (2nd ed.) , 1986 .

[13]  R. Haynes,et al.  Optimal search strategies for retrieving systematic reviews from Medline: analytical survey , 2004, BMJ : British Medical Journal.

[14]  Ping Zhang,et al.  Computer modeling of diabetes and its complications: a report on the Fifth Mount Hood challenge meeting. , 2007, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[15]  F. Dominici,et al.  Reproducible epidemiologic research. , 2006, American journal of epidemiology.

[16]  K. Baggerly Disclose all data in publications. , 2010, Nature.

[17]  Mirjam Kretzschmar,et al.  Dynamic Transmission Modeling , 2012, Medical decision making : an international journal of the Society for Medical Decision Making.

[18]  Uwe Siebert,et al.  Modeling good research practices--overview: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force--1. , 2012, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[19]  David M. Eddy,et al.  Archimedes: a new model for simulating health care systems--the mathematical formulation , 2002, J. Biomed. Informatics.

[20]  E. Long International Society for Pharmacoeconomics and Outcomes Research , 2012 .

[21]  M Sculpher,et al.  Review of guidelines for good practice in decision-analytic modelling in health technology assessment. , 2004, Health technology assessment.

[22]  Douglas K Owens,et al.  Analytic Tools for Public Health Decision Making , 2002, Medical decision making : an international journal of the Society for Medical Decision Making.

[23]  R. Haynes,et al.  Medline : analytical survey scientifically strong studies of diagnosis from Optimal search strategies for retrieving , 2004 .

[24]  Jonathan Karnon,et al.  Model Parameter Estimation and Uncertainty: a Report of the Ispor-smdm Modeling Good Research Practices Task Force-6 Background to the Task Forcemodel-parameter-estimation-and- Uncertainty-analysis.asp). a Summary of These Articles Was Pre- Sented at a Plenary Session at the Ispor 16th Annual Intern , 2022 .

[25]  R. Peng Reproducible Research in Computational Science , 2011, Science.

[26]  Paul Bratley,et al.  A guide to simulation , 1983 .

[27]  D M Eddy,et al.  The frequency of cervical cancer screening. Comparison of a mathematical model with empirical data , 1987, Cancer.

[28]  Natasha K. Stout,et al.  Calibration Methods Used in Cancer Simulation Models and Suggested Reporting Guidelines , 2012, PharmacoEconomics.

[29]  Anya Okhmatovskaia,et al.  Validation of population-based disease simulation models: a review of concepts and methods , 2010, BMC public health.

[30]  M. Drummond,et al.  Standardizing Methodologies for Economic Evaluation in Health Care: Practice, Problems, and Potential , 1993, International Journal of Technology Assessment in Health Care.

[31]  Patricia Scott,et al.  Screening Surgeons for HIV Infection: A Cost-effectiveness Analysis , 1995, Annals of Internal Medicine.

[32]  Niels Keiding,et al.  Reproducible research and the substantive context. , 2010, Biostatistics.

[33]  Murray Krahn,et al.  Conceptualizing a model: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force--2. , 2012, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[34]  Roger McHaney Computer Simulation: A Practical Perspective , 1991 .

[35]  Natasha K Stout,et al.  Empirically evaluating decision-analytic models. , 2010, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[36]  Jonathan Karnon,et al.  Model Parameter Estimation and Uncertainty Analysis , 2012, Medical decision making : an international journal of the Society for Medical Decision Making.

[37]  John Hornberger,et al.  Computer Modeling of Diabetes and Its Complications , 2007, Diabetes Care.

[38]  Roger D Peng,et al.  Reproducible research and Biostatistics. , 2009, Biostatistics.

[39]  C. Deangelis,et al.  The importance of independent academic statistical analysis. , 2010, Biostatistics.

[40]  Amy B. Knudsen,et al.  Evaluating Test Strategies for Colorectal Cancer Screening: A Decision Analysis for the U.S. Preventive Services Task Force , 2008, Annals of Internal Medicine.

[41]  S. Goodman,et al.  Reproducible Research: Moving toward Research the Public Can Really Trust , 2007, Annals of Internal Medicine.

[42]  J B Wong,et al.  Analytic Choices in Economic Models of Treatments for Rheumatoid Arthritis: What Makes a Difference? , 2005, Medical decision making : an international journal of the Society for Medical Decision Making.

[43]  Karl Claxton,et al.  Assessing Quality in Decision Analytic Cost-Effectiveness Models , 2000, PharmacoEconomics.

[44]  Jennifer M. Urban,et al.  Shining Light into Black Boxes , 2012, Science.

[45]  F Reed Johnson,et al.  Modeling for health care and other policy decisions: uses, roles, and validity. , 2002, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[46]  Milton C Weinstein,et al.  When to start antiretroviral therapy in resource-limited settings. , 2009, Annals of internal medicine.

[47]  Milton C Weinstein,et al.  Principles of good practice for decision analytic modeling in health-care evaluation: report of the ISPOR Task Force on Good Research Practices--Modeling Studies. , 2003, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[48]  Mirjam Kretzschmar,et al.  Dynamic Transmission Modeling: A Report of the ISPOR-SMDM Modeling Good Research Practices Task Force-5 , 2012, Value in Health.

[49]  TRISH GROVES The wider concept of data sharing: view from the BMJ. , 2010, Biostatistics.

[50]  Linus Schrage,et al.  A guide to simulation , 1983 .

[51]  D. Owens,et al.  State-transition modeling: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force--3. , 2012, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[52]  J D Graham,et al.  Modeling for health care and other policy decisions: uses, roles, and validity. , 2001, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[53]  J. Caro,et al.  Modeling Using Discrete Event Simulation , 2012, Medical decision making : an international journal of the Society for Medical Decision Making.

[54]  Averill M. Law,et al.  Simulation Modeling and Analysis , 1982 .

[55]  Justin Feigelman,et al.  Age at initiation and frequency of screening to detect type 2 diabetes: a cost-effectiveness analysis , 2010, The Lancet.

[56]  Elena Losina,et al.  Use of Genotypic Resistance Testing To Guide HIV Therapy: Clinical Impact and Cost-Effectiveness , 2001, Annals of Internal Medicine.

[57]  E Goyder,et al.  A review and critique of modelling in prioritising and designing screening programmes. , 2007, Health technology assessment.

[58]  G. Boriani,et al.  Cost-effectiveness of implantable cardioverter-defibrillators. , 2001, European heart journal.

[59]  S. Golder,et al.  Good Practice Guidelines for Decision-Analytic Modelling in Health Technology Assessment , 2012, PharmacoEconomics.

[60]  Arne Thesen,et al.  Simulation for Decision Making , 1992 .