Predictive-Adjusted Indirect Comparison: A Novel Method for Population Adjustment with Limited Access to Patient-Level Data

Population adjustment methods such as matching-adjusted indirect comparison (MAIC) are increasingly used to compare marginal treatment effects when there are cross-trial differences in effect modifiers and limited patient-level data. MAIC is based on propensity score weighting, which is sensitive to poor covariate overlap because of its inability to extrapolate. Current regression adjustment methods can extrapolate beyond the observed covariate space but target conditional treatment effects. This is problematic when the measure of effect is non-collapsible. To overcome these limitations, we develop a novel method based on multiple imputation called predictive-adjusted indirect comparison (PAIC). The novelty of PAIC is that it is a regression adjustment method that targets marginal treatment effects. It proceeds by splitting the adjustment into two separate stages: the generation of synthetic datasets and their analysis. We compare two versions of PAIC to MAIC in a comprehensive simulation study of 162 scenarios. This simulation study is based on binary outcomes and binary covariates and uses the log-odds ratio as the measure of effect. The simulation scenarios vary the trial sample size, prognostic variable effects, interaction effects, covariate correlations and covariate overlap. Generally, both PAIC and MAIC yield unbiased treatment effect estimates and valid coverage rates. In the simulations, PAIC provides more precise and more accurate estimates than MAIC, particularly when overlap is poor. MAIC and PAIC use different adjustment mechanisms and considering their results jointly may be helpful to evaluate the robustness of analyses.

[1]  Roland R. Ramsahai,et al.  From sample average treatment effect to population average treatment effect on the treated: combining experimental with observational studies to estimate population treatment effects , 2015 .

[2]  Elias Bareinboim,et al.  External Validity: From Do-Calculus to Transportability Across Populations , 2014, Probabilistic and Causal Inference.

[3]  David M. Phillippo Calibration of treatment effects in network meta-analysis using individual patient data , 2019 .

[4]  Jiqiang Guo,et al.  Stan: A Probabilistic Programming Language. , 2017, Journal of statistical software.

[5]  Bill Ravens,et al.  An Introduction to Copulas , 2000, Technometrics.

[6]  Martha Sajatovic,et al.  Clinical Prediction Models , 2013 .

[7]  Douglas G Altman,et al.  Validity of indirect comparison for estimating efficacy of competing interventions: empirical evidence from published meta-analyses , 2003, BMJ : British Medical Journal.

[8]  D. Madigan,et al.  Model Selection and Accounting for Model Uncertainty in Graphical Models Using Occam's Window , 1994 .

[9]  Joseph Kang,et al.  Demystifying Double Robustness: A Comparison of Alternative Strategies for Estimating a Population Mean from Incomplete Data , 2007, 0804.2958.

[10]  Harlan M Krumholz,et al.  Increasing value and reducing waste: addressing inaccessible research , 2014, The Lancet.

[11]  J. Pearl,et al.  Confounding and Collapsibility in Causal Inference , 1999 .

[12]  Robin Mitra,et al.  On the Use of Cauchy Prior Distributions for Bayesian Logistic Regression , 2015, Bayesian Analysis.

[13]  Keith Abrams,et al.  Use of Indirect and Mixed Treatment Comparisons for Technology Assessment , 2012, PharmacoEconomics.

[14]  K. Jack Ishak,et al.  No Head-to-Head Trial? Simulate the Missing Arms , 2012, PharmacoEconomics.

[15]  L. Cox Statistical Disclosure Limitation , 2006 .

[16]  Jennifer Hill,et al.  Assessing Methods for Generalizing Experimental Impact Estimates to Target Populations , 2016, Journal of research on educational effectiveness.

[17]  Paul Voigt,et al.  The EU General Data Protection Regulation (GDPR) , 2017 .

[18]  Stephen R Cole,et al.  Adjusted survival curves with inverse probability weights , 2004, Comput. Methods Programs Biomed..

[20]  Charles E McCulloch,et al.  Relaxing the rule of ten events per variable in logistic and Cox regression. , 2007, American journal of epidemiology.

[21]  P. Rosenbaum Model-Based Direct Adjustment , 1987 .

[22]  David M. Phillippo,et al.  Population Adjustment Methods for Indirect Comparisons: A Review of National Institute for Health and Care Excellence Technology Appraisals , 2019, International Journal of Technology Assessment in Health Care.

[23]  Nicky J Welton,et al.  NICE DSU Technical Support Document 18: Methods for population-adjusted indirect comparisons in submissions to NICE , 2016 .

[24]  S S Ellenberg,et al.  Placebo-Controlled Trials and Active-Control Trials in the Evaluation of New Treatments. Part 1: Ethical and Scientific Issues , 2000, Annals of Internal Medicine.

[25]  Miguel A. Juárez,et al.  Model-Based Clustering of Non-Gaussian Panel Data Based on Skew-t Distributions , 2010 .

[26]  David J. Lunn,et al.  The BUGS Book: A Practical Introduction to Bayesian Analysis , 2013 .

[27]  Alan Brnabic,et al.  Inclusion Of Multiple Studies In Matching Adjusted Indirect Comparisons (Maic) , 2015 .

[28]  David E. Booth,et al.  Analysis of Incomplete Multivariate Data , 2000, Technometrics.

[29]  S D Walter,et al.  The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. , 1997, Journal of clinical epidemiology.

[30]  W. Wien Econometric Computing with HC and HAC Covariance Matrix Estimators , 2004 .

[31]  Peter Dalgaard,et al.  R Development Core Team (2010): R: A language and environment for statistical computing , 2010 .

[32]  Mei Lu,et al.  Comparative Efficacy of Guanfacine Extended Release Versus Atomoxetine for the Treatment of Attention-Deficit/Hyperactivity Disorder in Children and Adolescents: Applying Matching-Adjusted Indirect Comparison Methodology , 2013, CNS Drugs.

[33]  Jerome P. Reiter,et al.  The Multiple Adaptations of Multiple Imputation , 2007 .

[34]  H. Chipman,et al.  Bayesian Additive Regression Trees , 2006 .

[35]  James R Carpenter,et al.  Sensitivity analysis after multiple imputation under missing at random: a weighting approach , 2007, Statistical methods in medical research.

[36]  T. Lanitis,et al.  Simulated Treatment Comparison of Time-To-Event (And Other Non-Linear) Outcomes. , 2015, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[37]  Christian P. Robert,et al.  Better together? Statistical learning in models made of modules , 2017, 1708.08719.

[38]  P. Austin An Introduction to Propensity Score Methods for Reducing the Effects of Confounding in Observational Studies , 2011, Multivariate behavioral research.

[39]  M. Sklar Fonctions de repartition a n dimensions et leurs marges , 1959 .

[40]  John K Kruschke,et al.  Bayesian data analysis. , 2010, Wiley interdisciplinary reviews. Cognitive science.

[41]  Martyn Plummer,et al.  JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling , 2003 .

[42]  Stef van Buuren,et al.  MICE: Multivariate Imputation by Chained Equations in R , 2011 .

[43]  Gianluca Baio,et al.  Bayesian Methods in Health Economics , 2012 .

[44]  J. Schafer,et al.  A comparison of inclusive and restrictive strategies in modern missing data procedures. , 2001, Psychological methods.

[45]  Zhiwei Zhang,et al.  Covariate-Adjusted Putative Placebo Analysis in Active-Controlled Clinical Trials , 2009 .

[46]  M. El-Masri Odds ratio. , 2013, The Canadian nurse.

[47]  Elizabeth A Stuart,et al.  Matching methods for causal inference: A review and a look forward. , 2010, Statistical science : a review journal of the Institute of Mathematical Statistics.

[48]  S Greenland,et al.  Interpretation and choice of effect measures in epidemiologic analyses. , 1987, American journal of epidemiology.

[49]  David M. Phillippo,et al.  Equivalence of entropy balancing and the method of moments for matching‐adjusted indirect comparison , 2020, Research synthesis methods.

[50]  Brian K. Lee,et al.  Weight Trimming and Propensity Score Weighting , 2011, PloS one.

[51]  Donald B. Rubin,et al.  Logit-Based Interval Estimation for Binomial Data Using the Jeffreys Prior , 1987 .

[52]  Gianluca Baio,et al.  Methods for Population Adjustment with Limited Access to Individual Patient Data: A Review and Simulation Study. , 2021, Research synthesis methods.

[53]  Eric Q. Wu,et al.  Comparative Effectiveness Without Head-to-Head Trials , 2012, PharmacoEconomics.

[54]  H. White A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity , 1980 .

[55]  F. Dominici,et al.  On quantifying the magnitude of confounding. , 2010, Biostatistics.

[56]  Gianluca Baio,et al.  A full Bayesian model to handle structural ones and missingness in economic evaluations from individual‐level data , 2018, Statistics in medicine.

[57]  Gerta Rücker,et al.  Presenting simulation results in a nested loop plot , 2014, BMC Medical Research Methodology.

[58]  Alan Brnabic,et al.  Alternative Weighting Approaches for Anchored Matching-Adjusted Indirect Comparisons via a Common Comparator. , 2019, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[59]  Bradley P. Carlin,et al.  Bayesian measures of model complexity and fit , 2002 .

[60]  Jerome P. Reiter,et al.  A Comparison of Posterior Simulation and Inference by Combining Rules for Multiple Imputation , 2011 .

[61]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[62]  Jerome P. Reiter,et al.  Multiple Imputation for Statistical Disclosure Limitation , 2003 .

[63]  J Lellouch,et al.  Efficiency of the logistic regression and Cox proportional hazards models in longitudinal studies. , 1989, Statistics in medicine.

[64]  A. Gelman Prior distributions for variance parameters in hierarchical models (comment on article by Browne and Draper) , 2004 .

[65]  Gianluca Baio,et al.  Methods for Population Adjustment with Limited Access to Individual Patient Data: A Simulation Study , 2020 .

[66]  G. Imbens,et al.  Estimation of Causal Effects using Propensity Score Weighting: An Application to Data on Right Heart Catheterization , 2001, Health Services and Outcomes Research Methodology.

[67]  Jerome P. Reiter,et al.  Releasing multiply imputed, synthetic public use microdata: an illustration and empirical study , 2005 .

[68]  Gillian M. Raab,et al.  Practical Data Synthesis for Large Samples , 2018, J. Priv. Confidentiality.

[69]  Sofia Dias,et al.  Methods for Population-Adjusted Indirect Comparisons in Health Technology Appraisal , 2017, Medical decision making : an international journal of the Society for Medical Decision Making.

[70]  D. Mark,et al.  Clinical prediction models: are we building better mousetraps? , 2003, Journal of the American College of Cardiology.

[71]  Nicholas R Latimer,et al.  Survival Analysis for Economic Evaluations Alongside Clinical Trials—Extrapolation with Patient-Level Data , 2013, Medical decision making : an international journal of the Society for Medical Decision Making.

[72]  Kenneth Stanley,et al.  Design of Randomized Controlled Trials , 2007, Circulation.

[73]  Irina Proskorovsky,et al.  Simulation and Matching-Based Approaches for Indirect Comparison of Treatments , 2015, PharmacoEconomics.

[74]  S. Senn,et al.  Stratification for the propensity score compared with linear regression techniques to assess the effect of treatment or exposure , 2007, Statistics in medicine.

[75]  H. Chipman,et al.  BART: Bayesian Additive Regression Trees , 2008, 0806.3286.

[76]  Lena Osterhagen,et al.  Multiple Imputation For Nonresponse In Surveys , 2016 .

[77]  C. Walsh,et al.  Assessing the impact of a matching‐adjusted indirect comparison in a Bayesian network meta‐analysis , 2019, Research synthesis methods.

[78]  Harald Binder,et al.  Recovery of original individual person data (IPD) inferences from empirical IPD summaries only: Applications to distributed computing under disclosure constraints , 2020, Statistics in medicine.

[79]  J. Pearl Causal diagrams for empirical research , 1995 .

[80]  N. Freemantle,et al.  The Effects of Model Misspecification in Unanchored Matching-Adjusted Indirect Comparison: Results of a Simulation Study. , 2020, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[81]  James R Carpenter,et al.  Sensitivity analysis for clinical trials with missing continuous outcome data using controlled multiple imputation: A practical guide , 2020, Statistics in medicine.

[82]  Alex J. Sutton,et al.  Evidence Synthesis for Decision Making 2: A Generalized Linear Modeling Framework for Pairwise and Network Meta-analysis of Randomized Controlled Trials. , 2013 .

[83]  J. Rahnenführer,et al.  Evaluation of Adjusted and Unadjusted Indirect Comparison Methods in Benefit Assessment , 2017, Methods of Information in Medicine.

[84]  P. Austin The use of propensity score methods with survival or time-to-event outcomes: reporting measures of effect similar to those used in randomized experiments , 2013, Statistics in medicine.

[85]  Wei-Yin Loh,et al.  Classification and regression trees , 2011, WIREs Data Mining Knowl. Discov..

[86]  Xiao-Li Meng,et al.  Multiple-Imputation Inferences with Uncongenial Sources of Input , 1994 .

[87]  M. Sculpher,et al.  Probabilistic sensitivity analysis for NICE technology assessment: not an optional extra. , 2005, Health economics.

[88]  Michael J Crowther,et al.  Using simulation studies to evaluate statistical methods , 2017, Statistics in medicine.

[89]  B. Giraudeau,et al.  Propensity score methods for estimating relative risks in cluster randomized trials with low‐incidence binary outcomes and selection bias , 2014, Statistics in medicine.

[90]  S. Cole,et al.  Generalizing evidence from randomized clinical trials to target populations: The ACTG 320 trial. , 2010, American journal of epidemiology.

[91]  Harold I Feldman,et al.  Individual patient‐ versus group‐level data meta‐regressions for the investigation of treatment effect modifiers: ecological bias rears its ugly head , 2002, Statistics in medicine.

[92]  Gary King,et al.  Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference , 2007, Political Analysis.

[93]  Gillian M. Raab,et al.  synthpop: Bespoke Creation of Synthetic Data in R , 2016 .

[94]  P. Trueman,et al.  ‘Fourth hurdle reviews’, NICE, and database applications , 2001, Pharmacoepidemiology and drug safety.

[95]  D G Altman,et al.  Indirect comparisons of competing interventions. , 2005, Health technology assessment.

[96]  John K. Kruschke,et al.  Doing Bayesian Data Analysis: A Tutorial with R, JAGS, and Stan , 2014 .

[97]  Nicky J Welton,et al.  Enhanced secondary analysis of survival data: reconstructing the data from published Kaplan-Meier survival curves , 2012, BMC Medical Research Methodology.

[98]  Andrea Manca,et al.  NICE DSU TECHNICAL SUPPORT DOCUMENT 17: THE USE OF OBSERVATIONAL DATA TO INFORM ESTIMATES OF TREATMENT EFFECTIVENESS IN TECHNOLOGY APPRAISAL: METHODS FOR COMPARATIVE INDIVIDUAL PATIENT DATA , 2015 .

[99]  Jerome P. Reiter,et al.  Satisfying Disclosure Restrictions With Synthetic Data Sets , 2002 .

[100]  D. Altman,et al.  The odds ratio , 2000, BMJ : British Medical Journal.

[101]  R. Bender,et al.  Methodological approach to determine minor, considerable, and major treatment effects in the early benefit assessment of new drugs , 2015, Biometrical journal. Biometrische Zeitschrift.

[102]  David M. Phillippo,et al.  Multilevel network meta‐regression for population‐adjusted treatment comparisons , 2020, Journal of the Royal Statistical Society. Series A,.

[103]  M. G. Pittau,et al.  A weakly informative default prior distribution for logistic and other regression models , 2008, 0901.4011.

[104]  Kurt Hornik,et al.  On the generation of correlated artificial binary data , 1998 .