Measurement error and timing of predictor values for multivariable risk prediction models are poorly reported.

OBJECTIVE Measurement error in predictor variables may threaten the validity of clinical prediction models. We sought to evaluate the possible extent of the problem. A secondary objective was to examine whether predictors are measured at the intended moment of model use. METHODS A systematic search of Medline was used to identify a sample of articles reporting the development of a clinical prediction model published in 2015. After screening according to a predefined inclusion criteria, information on predictors, strategies to control for measurement error, and intended moment of model use were extracted. Susceptibility to measurement error for each predictor was classified into low and high risks. RESULTS Thirty-three studies were reviewed, including 151 different predictors in the final prediction models. Fifty-one (33.7%) predictors were categorized as high risk of error; however, this was not accounted for in the model development. Only 8 (24.2%) studies explicitly stated the intended moment of model use and when the predictors were measured. CONCLUSION Reporting of measurement error and intended moment of model use is poor in prediction model studies. There is a need to identify circumstances where ignoring measurement error in prediction models is consequential and whether accounting for the error will improve the predictions.

[1]  I. Garutti,et al.  Early Measurement of Indocyanine Green Clearance Accurately Predicts Short-Term Outcomes After Liver Transplantation , 2016, Transplantation.

[2]  Etienne Cavalier,et al.  Normal reference values for glomerular filtration rate: what do we really know? , 2012, Nephrology, dialysis, transplantation : official publication of the European Dialysis and Transplant Association - European Renal Association.

[3]  R. Angioli,et al.  A Predictive Score for Secondary Cytoreductive Surgery in Recurrent Ovarian Cancer (SeC-Score): A Single-Centre, Controlled Study for Preoperative Patient Selection , 2015, Annals of Surgical Oncology.

[4]  Raymond J. Carroll,et al.  Measurement error in nonlinear models: a modern perspective , 2006 .

[5]  R. Soslow,et al.  Difficulties in Assessing the Depth of Myometrial Invasion in Endometrial Carcinoma , 2007, International journal of gynecological pathology : official journal of the International Society of Gynecological Pathologists.

[6]  Nada Esa,et al.  Addition of B-Type Natriuretic Peptide to Existing Clinical Risk Scores Enhances Identification of Patients at Risk for Atrial Fibrillation Recurrence After Pulmonary Vein Isolation. , 2015, Critical pathways in cardiology.

[7]  Annamaria Guolo,et al.  Robust techniques for measurement error correction: a review , 2008, Statistical methods in medical research.

[8]  G. Collins,et al.  Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies: The CHARMS Checklist , 2014, PLoS medicine.

[9]  X. Tan,et al.  Multivariate logistic regression analysis of postoperative complications and risk model establishment of gastrectomy for gastric cancer: A single-center cohort report , 2016, Scandinavian journal of gastroenterology.

[10]  J. Handler,et al.  The importance of accurate blood pressure measurement. , 2009, The Permanente journal.

[11]  T. Choueiri,et al.  Risk factors and model for predicting toxicity‐related treatment discontinuation in patients with metastatic renal cell carcinoma treated with vascular endothelial growth factor–targeted therapy: Results from the International Metastatic Renal Cell Carcinoma Database Consortium , 2016, Cancer.

[12]  Gary S Collins,et al.  Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): Explanation and Elaboration , 2015, Annals of Internal Medicine.

[13]  Justin M Ream,et al.  Prostate Cancer: Utility of Whole-Lesion Apparent Diffusion Coefficient Metrics for Prediction of Biochemical Recurrence After Radical Prostatectomy. , 2015, AJR. American journal of roentgenology.

[14]  Richard D Riley,et al.  Prognosis research strategy (PROGRESS) 1: A framework for researching clinical outcomes , 2013, BMJ : British Medical Journal.

[15]  T. Masaki,et al.  Unavoidable Human Errors of Tumor Size Measurement during Specimen Attachment after Endoscopic Resection: A Clinical Prospective Study , 2015, PloS one.

[16]  B. Wessler,et al.  Editorial See P 332 , 2022 .

[17]  A. Hill,et al.  Body mass index: a comparison between self-reported and measured height and weight. , 1998, Journal of public health medicine.

[18]  Painful Memories: Reliability of Pain Intensity Recall at 3 Months in Senior Patients , 2017, Pain research & management.

[19]  R. Prentice Covariate measurement errors and parameter estimation in a failure time regression model , 1982 .

[20]  M. Ferreira,et al.  Back Complaints in the Elders (BACE); design of cohort studies in primary care: an international consortium , 2011, BMC musculoskeletal disorders.

[21]  G. Collins,et al.  Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): The TRIPOD Statement , 2015, Annals of Internal Medicine.

[22]  P. Winkel,et al.  Factors contributing to intra-individual variation of serum constituents: 5. Short-term day-to-day and within-hour variation of serum constituents in healthy subjects. , 1974, Clinical chemistry.

[23]  G. Mancia,et al.  Total Cardiovascular Risk, Blood Pressure Variability and Adrenergic Overdrive in Hypertension: Evidence, Mechanisms and Clinical Implications , 2012, Current Hypertension Reports.

[24]  Carl van Walraven,et al.  Time-dependent bias was common in survival analyses published in leading clinical journals. , 2004, Journal of clinical epidemiology.

[25]  J. Reeves,et al.  CRISP: Catheterization RISk score for pediatrics: A Report from the Congenital Cardiac Interventional Study Consortium (CCISC) , 2016, Catheterization and cardiovascular interventions : official journal of the Society for Cardiac Angiography & Interventions.

[26]  P. Gustafson Measurement Error and Misclassification in Statistics and Epidemiology: Impacts and Bayesian Adjustments , 2003 .

[27]  C. Cullum,et al.  A Preliminary Formula to Predict Timing of Symptom Resolution for Collegiate Athletes Diagnosed With Sport Concussion. , 2015, Journal of athletic training.

[28]  Ying Sun,et al.  Neoadjuvant chemotherapy in locally advanced nasopharyngeal carcinoma: Defining high-risk patients who may benefit before concurrent chemotherapy combined with intensity-modulated radiotherapy , 2015, Scientific Reports.

[29]  M. Numans,et al.  Identification of patients at risk for colorectal cancer in primary care: an explorative study with routine healthcare data , 2015, European journal of gastroenterology & hepatology.

[30]  Mauro Panteghini,et al.  Generation of data on within-subject biological variation in laboratory medicine: An update , 2016, Critical reviews in clinical laboratory sciences.

[31]  Sabine Maguire,et al.  Methodological standards for the development and evaluation of clinical prediction rules: a review of the literature , 2019, Diagnostic and Prognostic Research.

[32]  R. Tracy,et al.  Variability in the measurement of C-reactive protein in healthy subjects: implications for reference intervals and epidemiological applications. , 1997, Clinical chemistry.

[33]  Guangyu Liu,et al.  Predicting Non-sentinel Lymph Node Metastasis in a Chinese Breast Cancer Population with 1–2 Positive Sentinel Nodes: Development and Assessment of a New Predictive Nomogram , 2015, World Journal of Surgery.

[34]  G. Spolverato,et al.  Can hepatic resection provide a long‐term cure for patients with intrahepatic cholangiocarcinoma? , 2015, Cancer.

[35]  C. Fife,et al.  A Predictive Model for Pressure Ulcer Outcome: The Wound Healing Index , 2015, Advances in skin & wound care.

[36]  R. Deyo,et al.  Back pain in primary care. Outcomes at 1 year. , 1993, Spine.

[37]  A. Sapino,et al.  Elaboration of a nomogram to predict non sentinel node status in breast cancer patients with positive sentinel node, intra-operatively assessed with one step nucleic acid amplification method , 2015, Journal of Experimental & Clinical Cancer Research.

[38]  Richard D Riley,et al.  Prognosis research ideally should measure time-varying predictors at their intended moment of use , 2017, Diagnostic and Prognostic Research.

[39]  L. Sawers,et al.  Measuring and modelling concurrency , 2013, Journal of the International AIDS Society.

[40]  Maarten van Smeden,et al.  Measurement error is often neglected in medical literature: a systematic review. , 2018, Journal of clinical epidemiology.

[41]  M. Silverberg,et al.  A validated web‐based tool to display individualised Crohn's disease predicted outcomes based on clinical, serologic and genetic variables , 2016, Alimentary pharmacology & therapeutics.

[42]  Malka Gorfine,et al.  The impact of covariate measurement error on risk prediction , 2015, Statistics in medicine.

[43]  P. Elson,et al.  The "real-life" variability of CA-125 in ovarian cancer patients. , 2006, Gynecologic oncology.

[44]  Hiromitsu Kobayashi,et al.  Effect of measurement duration on accuracy of pulse-counting , 2013, Ergonomics.

[45]  G. Morgia,et al.  Performance of biopsy factors in predicting unfavorable disease in patients eligible for active surveillance according to the PRIAS criteria , 2015, Prostate Cancer and Prostatic Disease.

[46]  A. Chiang,et al.  A joint model based on longitudinal CA125 in ovarian cancer to predict recurrence. , 2016, Biomarkers in medicine.

[47]  Mardge H. Cohen,et al.  Gender-Related Risk Factors Improve Mortality Predictive Ability of VACS Index Among HIV-Infected Women , 2015, Journal of acquired immune deficiency syndromes.

[48]  E W Steyerberg,et al.  Impact of predictor measurement heterogeneity across settings on the performance of prediction models: A measurement error perspective , 2019, Statistics in medicine.

[49]  S. Desai,et al.  Development of a scoring system to estimate mortality in abdominal aortic aneurysms management , 2015, Vascular.

[50]  K. Wardenaar,et al.  DATA‐DRIVEN COURSE TRAJECTORIES IN PRIMARY CARE PATIENTS WITH MAJOR DEPRESSIVE DISORDER , 2014, Depression and anxiety.

[51]  Wei Li,et al.  A resampling approach for adjustment in prediction models for covariate measurement error , 2005, Comput. Methods Programs Biomed..

[52]  P. H. Petersen,et al.  Assessment of biological variation and analytical imprecision of CA 125, CEA, and TPA in relation to monitoring of ovarian cancer. , 1999, Gynecologic oncology.

[53]  J. Hux,et al.  The influence of measurement error on calibration, discrimination, and overall estimation of a risk prediction model , 2012, Population Health Metrics.

[54]  J. Allenberg,et al.  Prognostic value of a low post-exercise ankle brachial index as assessed by primary care physicians. , 2011, Atherosclerosis.

[55]  J. Takanashi,et al.  Predictive score for early diagnosis of acute encephalopathy with biphasic seizures and late reduced diffusion (AESD) , 2015, Journal of the Neurological Sciences.

[56]  Ruth Etzioni,et al.  Precision Medicine in Active Surveillance for Prostate Cancer: Development of the Canary-Early Detection Research Network Active Surveillance Biopsy Risk Calculator. , 2015, European urology.

[57]  S. Greenland,et al.  Proper interpretation of non-differential misclassification effects: expectations vs observations. , 2005, International journal of epidemiology.

[58]  Shaoxing Yang,et al.  Prediction of survival prognosis after surgery in patients with symptomatic metastatic spinal cord compression from non-small cell lung cancer , 2015, BMC Cancer.

[59]  M. Deraco,et al.  The Role of Ki-67 and Pre-cytoreduction Parameters in Selecting Diffuse Malignant Peritoneal Mesothelioma (DMPM) Patients for Cytoreductive Surgery (CRS) and Hyperthermic Intraperitoneal Chemotherapy (HIPEC) , 2016, Annals of Surgical Oncology.

[60]  Shuai Leng,et al.  Small (< 4 cm) Renal Masses: Differentiation of Angiomyolipoma Without Visible Fat From Renal Cell Carcinoma Using Unenhanced and Contrast-Enhanced CT. , 2015, AJR. American journal of roentgenology.

[61]  M. Peake,et al.  Measurement of serum creatinine--current status and future goals. , 2006, The Clinical biochemist. Reviews.

[62]  Uri Kartoun,et al.  Development and Validation of an Algorithm to Identify Nonalcoholic Fatty Liver Disease in the Electronic Medical Record , 2016, Digestive Diseases and Sciences.

[63]  M. Leeflang,et al.  Search Filters for Finding Prognostic and Diagnostic Prediction Studies in Medline to Enhance Systematic Reviews , 2012, PloS one.

[64]  D. A. van der Windt,et al.  Functional outcome in older adults with joint pain and comorbidity: design of a prospective cohort study , 2011, BMC musculoskeletal disorders.

[65]  E. Erlandsen,et al.  Biological variation of cystatin C and creatinine , 2009, Scandinavian journal of clinical and laboratory investigation.

[66]  E. Steyerberg,et al.  Prognosis Research Strategy (PROGRESS) 3: Prognostic Model Research , 2013, PLoS medicine.

[67]  G. Maurer,et al.  History of previous bleeding and C-reactive protein improve assessment of bleeding risk in elderly patients (≥80 years) with myocardial infarction , 2015, Thrombosis and Haemostasis.

[68]  S. Sherman,et al.  Development and Validation of a Prediction Model for Admission After Endoscopic Retrograde Cholangiopancreatography. , 2015, Clinical gastroenterology and hepatology : the official clinical practice journal of the American Gastroenterological Association.

[69]  C. Coutant,et al.  A Predictive Model Using Histopathologic Characteristics of Early-Stage Type 1 Endometrial Cancer to Identify Patients at High Risk for Lymph Node Metastasis , 2015, Annals of Surgical Oncology.

[70]  S. Roman,et al.  A Bedside Risk Calculator to Preoperatively Distinguish Follicular Thyroid Carcinoma from Follicular Variant of Papillary Thyroid Carcinoma , 2015, World Journal of Surgery.

[71]  Jean-Paul Fox,et al.  Bayesian modeling of measurement error in predictor variables using item response theory , 2003 .

[72]  Sae Rom Hong,et al.  Prognostic value of SYNTAX score based on coronary computed tomography angiography. , 2015, International journal of cardiology.

[73]  E. Huskisson,et al.  Accuracy of subjective measurements made with or without previous scores: an important source of error in serial measurement of subjective states. , 1979, Annals of the rheumatic diseases.

[74]  K. Matsuo,et al.  Predictive Factor of Conversion to Laparotomy in Minimally Invasive Surgical Staging for Endometrial Cancer , 2015, International Journal of Gynecologic Cancer.

[75]  C. Roehrborn,et al.  Variability of repeated serum prostate-specific antigen (PSA) measurements within less than 90 days in a well-defined patient population. , 1996, Urology.

[76]  P. Taourel,et al.  Value of CT to predict surgically important bowel and/or mesenteric injury in blunt trauma: performance of a preliminary scoring system , 2015, European Radiology.

[77]  John M S Bartlett,et al.  An international Ki67 reproducibility study. , 2013, Journal of the National Cancer Institute.

[78]  M. Panteghini,et al.  The importance of individual biology in the clinical use of serum biomarkers for ovarian cancer , 2014, Clinical chemistry and laboratory medicine.

[79]  Jeffrey T. Howard,et al.  Toward a More Robust Prediction of Pulmonary Embolism in Trauma Patients: A Risk Assessment Model Based on 38,000 Patients , 2016, Journal of orthopaedic trauma.

[80]  B. Radanov,et al.  Role of psychosocial stress in recovery from common whiplash , 1991, The Lancet.

[81]  E. Steyerberg Clinical Prediction Models , 2008, Statistics for Biology and Health.

[82]  H. van Marwijk,et al.  The prognosis of undetected depression in older general practice patients. A one year follow-up study. , 2009, Journal of affective disorders.

[83]  R. Riley,et al.  Evidence synthesis in prognosis research , 2019, Diagnostic and Prognostic Research.