PROBAST: A Tool to Assess Risk of Bias and Applicability of Prediction Model Studies: Explanation and Elaboration

Prediction models in health care aim to predict for an individual whether a particular outcome, such as disease, is present (diagnostic models) or whether it will occur in the future (prognostic models) (16). Diagnostic models can be used to refer patients for further testing, to initiate treatment, or to inform patients. Prognostic models can be used to aid decisions about preventive lifestyle changes, therapeutic interventions, or monitoring strategies or to stratify risk in randomized trial design and analysis (7, 8). Potential users of prediction models include health care professionals, policymakers, guideline developers, patients, and the general public. The medical literature contains thousands of studies developing and validating prediction models and often has numerous prediction models for the same target population and outcomes. For example, more than 60 models address breast cancer prognosis (9), more than 250 exist in obstetrics (10), and nearly 800 predict outcomes in patients with cardiovascular disease (11). This proliferation of prediction models will increase further with the growth of personalized or precision medicine. Systematic reviews are considered the most reliable form of evidence when addressing randomized therapeutic studies and studies of diagnostic test accuracy (12). In the era of personalized and precision medicine, interest in systematic reviews of prediction model studies is rapidly growing, as exemplified by the formation of the Cochrane Prognosis Methods Group to support systematic reviews of prognosis, including prediction model studies (13, 14). Guidance to facilitate systematic reviews of prediction models has been developed (Table 1), including for search strategies (15, 4143), formulation of the review question (16, 17), data extraction (16), and meta-analysis (17, 2225, 40, 44, 45). Table 1. Guidance on Conducting Systematic Reviews of Prediction Model Studies Assessment of risk of bias (ROB) is an essential step in any systematic review. Shortcomings in study design, conduct, and analysis can result in study estimates being at ROBthat is, at risk of results being flawed or distorted. When interpreting results from a systematic review, readers can draw stronger conclusions from a review based on primary studies at low ROB than from one based on studies at high or unclear ROB (46). Identifying the studies most relevant to the settings and populations targeted in the review (based on the applicability of primary studies to the review question) is also important. We therefore developed PROBAST (Prediction model Risk Of Bias ASsessment Tool) to address the lack of suitable tools designed specifically to assess ROB and applicability of primary prediction model studies. PROBAST consists of 4 domains containing 20 signaling questions to facilitate ROB assessment (39). The structure and rating system are similar to those in tools designed to assess ROB in randomized trials (revised Cochrane ROB Tool [ROB 2.0]), diagnostic accuracy studies (QUADAS-2 [Quality Assessment of Diagnostic Accuracy Studies 2]), and systematic reviews (ROBIS) (37, 47, 48). Although PROBAST was designed for use in systematic reviews of prediction model studies, it can also be used as a general tool for critical appraisal of (primary) prediction model studies. Here, we describe the rationale behind the domains and signaling questions, how to use them, and how to reach domain-level and overall judgments about ROB and applicability of primary studies to a review question. At the Web site (www.probast.org), 5 filled-in examples from across the medical field illustrate these processes. Because this is an area of active research, the tool, examples, and accompanying guidance will be updated when needed, and the latest version of PROBAST should always be downloaded from the Web site. Focus of PROBAST PROBAST is designed to assess primary studies that develop, validate, or update (for example, extend) multivariable prediction models for diagnosis or prognosis (Boxes 1 and 2). A multivariable prediction model is defined as any combination or equation of 2 or more predictors (such as age, sex, symptoms, signs, disease stage, or biomarkers) for estimating for an individual the probability or risk of having (diagnosis) or developing (prognosis) a particular outcome (1, 4, 68, 49, 50). Other names for prediction model include risk prediction model, predictive model, prediction index or rule, and risk score (1, 38, 4951). Box 1. Types of diagnostic and prognostic modeling studies or reports addressed by PROBAST. Adopted from the TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) and CHARMS (CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies) guidance (8, 16). Box 2. Differences between diagnostic and prognostic prediction model studies. PROBAST = Prediction model Risk Of Bias ASsessment Tool. Diagnostic and Prognostic Models Diagnostic prediction models estimate the probability that a certain outcome, the target condition, is currently present. Diagnostic prediction model studies typically include individuals who are suspectedbut not yet knownto have the target condition. Prognostic prediction models estimate the probability that a future outcome or event will occur, such as death, disease recurrence, disease complication, or therapy response. The time period of prediction can vary from hours (for example, preoperatively predicting postoperative nausea and vomiting) to years (for example, predicting lifelong risk for a coronary event). Although many prognostic models enroll patients with an established diagnosis, this does not have to be the starting point, as seen in models for predicting development of diabetes in pregnant women (52) or of osteoporotic fractures in the general population (53). Consistent with the TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) statement (7, 8), PROBAST thus broadly defines prognostic models as those predicting a future outcome in persons at risk for that outcome. Diagnostic and prognostic model studies often use different terms for predictors and outcomes (Box 2). The cancer literature frequently distinguishes between prognostic and predictive models, such that predictive models identify individuals with differential treatment effects (54). These types of (predictive) models are outside the scope of this article. Types of Predictors, Outcomes, and Modeling Techniques PROBAST can be used to assess any type of diagnostic or prognostic prediction model aimed at individualized predictions regardless of the predictors used; outcomes being predicted; or methods used to develop, validate, or update (for example, extend) the model. Predictors range from demographic characteristics, medical history, and physical examination results; to imaging results, electrophysiology, blood, urine, or tissue measurements, and disease stages or characteristics; to results from omics and other new biological measurements. Predictors are also called covariates, risk indicators, prognostic factors, determinants, index test results, or independent variables (4, 68, 49, 50, 55, 56, 57). PROBAST distinguishes between candidate predictors and predictors included in the final model (57). Candidate predictors are variables considered potentially predictive of the outcome presence (diagnosis) or occurrence (prognosis)that is, all those evaluated in the study regardless of whether they are included in the final multivariable model. PROBAST primarily addresses prediction models for binary and time-to-event outcomes because these are the most common in medicine. However, the tool can also be used to assess models predicting nonbinary outcomes, such as continuous scores (for example, pain scores or cholesterol levels) or categorical outcomes (for example, the Glasgow Coma Scale). Almost all PROBAST signaling questions apply equally to continuous and categorical outcomes, except questions addressing number of outcome events per predictor and certain measures of model performance (such as the c-statistic), which are not relevant to continuous outcomes. Prediction models usually involve regression modeling techniques, such as logistic regression or survival models. Prediction models may also be developed or validated using nonregression techniques, such as neural networks, random forests, or support vector machines. As the use of routine care (and big) data increases, additional modeling techniques are becoming more common, including machine and artificial learning models. The main differences between studies using regression and other types of prediction modeling include the methods of data analysis; nonregression development models can often have greater risks of overfitting when data are sparse, and the potential lack of transparency can affect the applicability and usability of the models. In the section on tailoring PROBAST with additional signaling questions, we provide guidance about how PROBAST can be adapted to address other types of outcomes and modeling techniques. Types of Review Questions PROBAST can be used to assess different types of systematic review questions. For some review questions, all prediction model studies are relevant (including both development and validation), but for other questions only validation studies are relevant. Box 3 gives examples of potential review questions for both prognostic and diagnostic prediction models where PROBAST is applicable. CHARMS (CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies) and Table 2 provide explicit guidance on how to frame a focused question for reviews of prediction model studies (16, 17). Box 3. Examples of systematic review questions for which PROBAST is suitable. There are various different questions that systematic reviews of