Biomarker combinations for diagnosis and prognosis in multicenter studies: Principles and methods

Many investigators are interested in combining biomarkers to predict a binary outcome or detect underlying disease. This endeavor is complicated by the fact that many biomarker studies involve data from multiple centers. Depending upon the relationship between center, the biomarkers, and the target of prediction, care must be taken when constructing and evaluating combinations of biomarkers. We introduce a taxonomy to describe the role of center and consider how a biomarker combination should be constructed and evaluated. We show that ignoring center, which is frequently done by clinical researchers, is often not appropriate. The limited statistical literature proposes using random intercept logistic regression models, an approach that we demonstrate is generally inadequate and may be misleading. We instead propose using fixed intercept logistic regression, which appropriately accounts for center without relying on untenable assumptions. After constructing the biomarker combination, we recommend using performance measures that account for the multicenter nature of the data, namely the center-adjusted area under the receiver operating characteristic curve. We apply these methods to data from a multicenter study of acute kidney injury after cardiac surgery. Appropriately accounting for center, both in construction and evaluation, may increase the likelihood of identifying clinically useful biomarker combinations.

[1]  J. Kalbfleisch,et al.  Between- and within-cluster covariate effects in the analysis of clustered data. , 1998, Biometrics.

[2]  A. Garg,et al.  Postoperative biomarkers predict acute kidney injury and poor outcomes after adult cardiac surgery. , 2011, Journal of the American Society of Nephrology : JASN.

[3]  C. McCulloch,et al.  Misspecifying the Shape of a Random Effects Distribution: Why Getting It Wrong May Not Matter , 2011, 1201.1980.

[4]  Margaret Sullivan Pepe,et al.  Combining Several Screening Tests: Optimality of the Risk Score , 2002, Biometrics.

[5]  T. Cai,et al.  Combining Predictors for Classification Using the Area under the Receiver Operating Characteristic Curve , 2006, Biometrics.

[6]  J. Vermunt,et al.  A Comparison of Multilevel Logistic Regression Models with Parametric and Nonparametric Random Intercepts , 2008 .

[7]  T. Nickolas,et al.  Diagnostic and prognostic stratification in the emergency department using urinary biomarkers of nephron damage: a multicenter prospective cohort study. , 2012, Journal of the American College of Cardiology.

[8]  Daniel Cooley,et al.  Modelling pairwise dependence of maxima in space , 2009 .

[9]  Gareth Ambler,et al.  A note on obtaining correct marginal predictions from a random intercepts model for binary outcomes , 2015, BMC Medical Research Methodology.

[10]  Zhi Geng,et al.  Collapsibility of logistic regression coefficients , 1995 .

[11]  W Bouwmeester,et al.  Internal validation of risk models in clustered data: a comparison of bootstrap schemes. , 2013, American journal of epidemiology.

[12]  Yvonne Vergouwe,et al.  Interpretation of concordance measures for clustered data , 2014, Statistics in Medicine.

[13]  E. Lesaffre,et al.  An application of Harrell's C‐index to PH frailty models , 2010, Statistics in medicine.

[14]  Susan Weaver,et al.  LETTER TO THE EDITOR: ASSOCIATION MODELS FOR PERIODONTAL DISEASE PROGRESSION: A COMPARISON OF METHODS FOR CLUSTERED BINARY DATA by T. R. Ten Have, J. R. Landis and S. Weaver, Statistics in Medicine14, 413–430 (1995) , 1996 .

[15]  Zhulin He,et al.  Adjusting for confounding by cluster using generalized linear mixed models , 2010 .

[16]  B. Kahan Accounting for centre-effects in multicentre trials with a binary outcome – when, why, and how? , 2014, BMC Medical Research Methodology.

[17]  Charles E. McCulloch,et al.  Separating between‐ and within‐cluster covariate effects by using conditional and partitioning methods , 2006 .

[18]  A. Localio,et al.  Adjustments for Center in Multicenter Studies: An Overview , 2001, Annals of Internal Medicine.

[19]  Sunil J Rao,et al.  Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis , 2003 .

[20]  Patrick J Heagerty,et al.  On outcome-dependent sampling designs for longitudinal binary response data with time-varying covariates. , 2008, Biostatistics.

[21]  P. Bedossa,et al.  Diagnostic accuracy of FibroScan and comparison to liver fibrosis biomarkers in chronic viral hepatitis: a multicenter prospective study (the FIBROSTIC study). , 2010, Journal of hepatology.

[22]  Holly Janes,et al.  Adjusting for covariate effects on classification accuracy using the covariate-adjusted receiver operating characteristic curve. , 2009, Biometrika.

[23]  Andrew Copas,et al.  Review of methods for handling confounding by cluster and informative cluster size in clustered data , 2014, Statistics in medicine.

[24]  J. R. Landis,et al.  Population-averaged and cluster-specific models for clustered ordinal response data. , 1996, Statistics in medicine.

[25]  Yannan Jiang,et al.  Likelihood‐based analysis of longitudinal data from outcome‐related sampling designs , 2014, Biometrics.

[26]  Hao W Zheng,et al.  Adjusting for confounding by neighborhood using a proportional odds model and complex survey data. , 2012, American journal of epidemiology.

[27]  Sophia Rabe-Hesketh,et al.  Handling initial conditions and endogenous covariates in dynamic/transition models for binary data with unobserved heterogeneity , 2014 .

[28]  A. Feldstein,et al.  Cytokeratin‐18 fragment levels as noninvasive biomarkers for nonalcoholic steatohepatitis: A multicenter validation study , 2009, Hepatology.

[29]  J. Kalbfleisch,et al.  The effects of mixture distribution misspecification when fitting mixed-effects logistic models , 1992 .

[30]  M. Lesperance,et al.  Estimation efficiency in a binary mixed-effects model setting , 1996 .

[31]  D. Hochstrasser,et al.  Cardiac biomarkers for risk stratification in non‐massive pulmonary embolism: a multicenter prospective study , 2009, Journal of thrombosis and haemostasis : JTH.

[32]  Yvonne Vergouwe,et al.  Prediction models for clustered data: comparison of a random intercept and standard regression model , 2013, BMC Medical Research Methodology.

[33]  Michael K Parides,et al.  Separation of individual‐level and cluster‐level covariate effects in regression analysis of correlated data , 2003, Statistics in medicine.

[34]  J. Ioannidis Biomarker failures. , 2013, Clinical chemistry.

[35]  J. Neyman,et al.  Consistent Estimates Based on Partially Consistent Observations , 1948 .

[36]  J R Landis,et al.  Association models for periodontal disease progression: a comparison of methods for clustered binary data. , 1995, Statistics in medicine.

[37]  J. Copas,et al.  Overestimation of the receiver operating characteristic curve for logistic regression , 2002 .

[38]  L. Armstrong A Prospective Multicenter Derivation of a Biomarker Panel to Assess Risk of Organ Dysfunction, Shock, and Death in Emergency Department Patients with Suspected Sepsis , 2009 .

[39]  F B Hu,et al.  Comparison of population-averaged and subject-specific approaches for analyzing repeated binary outcomes. , 1998, American journal of epidemiology.

[40]  Gary Longton,et al.  Accommodating Covariates in Receiver Operating Characteristic Analysis , 2009 .

[41]  Zhehui Luo,et al.  Fixed effects, random effects and GEE: What are the differences? , 2009, Statistics in medicine.

[42]  P. Heagerty,et al.  Misspecified maximum likelihood estimates and generalised linear mixed models , 2001 .

[43]  T. Blakely,et al.  Fixed effects analysis of repeated measures data. , 2014, International journal of epidemiology.

[44]  T R Ten Have,et al.  An Empirical Comparison of Several Clustered Data Approaches Under Confounding Due to Cluster Effects in the Analysis of Complications of Coronary Angioplasty , 1999, Biometrics.

[45]  J. Pearl,et al.  Confounding and Collapsibility in Causal Inference , 1999 .

[46]  Scott L. Zeger,et al.  Marginalized Multilevel Models and Likelihood Inference , 2000 .

[47]  Holly Janes,et al.  Practice of Epidemiology Adjusting for Covariates in Studies of Diagnostic, Screening, or Prognostic Markers: an Old Concept in a New Setting , 2022 .

[48]  J. Kalbfleisch,et al.  A Comparison of Cluster-Specific and Population-Averaged Approaches for Analyzing Correlated Binary Data , 1991 .

[49]  Jesse A. Berlin,et al.  Confounding Due to Cluster in Multicenter Studies—Causes and Cures , 2002, Health Services and Outcomes Research Methodology.

[50]  Gary Longton,et al.  Accommodating Covariates in ROC Analysis. , 2009, The Stata journal.

[51]  M. Pepe The Statistical Evaluation of Medical Tests for Classification and Prediction , 2003 .