Evaluation of the Quality of Prognosis Studies in Systematic Reviews

Prognosis studies are investigations of future events or the evaluation of associations between risk factors and health outcomes in populations of patients (1). The results of such studies improve our understanding of the clinical course of a disease and assist clinicians in making informed decisions about how best to manage patients. Prognostic research also informs the design of intervention studies by helping define subgroups of patients who may benefit from a new treatment and by providing necessary information about the natural history of a disorder (2). There has recently been a rapid increase in the use of systematic review methods to synthesize the evidence on research questions related to prognosis. It is essential that investigators conducting systematic reviews thoroughly appraise the methodologic quality of included studies to be confident that a study's design, conduct, analysis, and interpretation have adequately reduced the opportunity for bias (3, 4). Caution is warranted, however, because inclusion of methodologically weak studies can threaten the internal validity of a systematic review (4). This follows abundant empirical evidence that inadequate attention to biases can cause invalid results and inferences (5-9). However, there is limited consensus on how to appraise the quality of prognosis studies (1). A useful framework to assess bias in such studies follows the basic principles of epidemiologic research (10, 11). We focus on 6 areas of potential bias: study participation, study attrition, prognostic factor measurement, confounding measurement and account, outcome measurement, and analysis. The main objectives of our review of reviews are to describe methods used to assess the quality of prognosis studies and to describe how well current practices assess potential biases. Our secondary objective is to develop recommendations to guide future quality appraisal, both within single studies of prognostic factors and within systematic reviews of the evidence. We hope this work facilitates future discussion and research on biases in prognosis studies and systematic reviews. Methods Literature Search and Study Selection We identified systematic reviews of prognosis studies by searching MEDLINE (1966 to October 2005) using the search strategy recommended by McKibbon and colleagues (12). This strategy combines broad search terms for systematic reviews (systematic review.mp; meta-analysis.mp) and a sensitive search strategy for prognosis studies (cohort, incidence, mortality, follow-up studies, prognos*, predict*, or course). We also searched the reference lists of included reviews and methodologic papers to identify other relevant publications. We restricted our search to English-language publications. One reviewer conducted the search and selected the studies. Systematic reviews, defined as reviews of published studies with a comprehensive search and systematic selection, were included if they assessed the methodologic quality of the included studies by using 1 or more explicit criteria. We excluded studies if they were meta-analyses of independent patient data only, if their primary goal was to investigate the effectiveness of an intervention or specific diagnostic or screening tests, or if they included studies that were not done on humans. Data Extraction and Synthesis Individual items included in the quality assessment of the systematic reviews were recorded as they were reported in the publication (that is, the information that would be available to readers and future reviewers). We reviewed journal Web sites and contacted the authors of the systematic reviews for additional information when authors made such an offer in their original papers. When reviews assessed different study designs by using different sets of quality items, we extracted only those items used to assess cohort studies. We constructed a comprehensive list of distinct items that the reviews used to assess the quality of their included studies. The full text of each review was screened. All items used by the review authors to assess the quality of studies were extracted into a computerized spreadsheet by 1 reviewer. Two experienced reviewers, a clinical epidemiologist and an epidemiologist, independently synthesized the quality items extracted from the prognosis reviews to determine how well the systematic reviews assessed potential biases. We did this in 3 steps: 1) identified distinct concepts or domains addressed by the quality items; 2) grouped each extracted quality item into the appropriate domain or domains; and 3) identified the domains necessary to assess potential biases in prognosis studies. We then used this information to assess how well the reviews' quality assessment included items from the domains necessary to assess potential biases. After completing each of the first 3 steps, the reviewers met to attempt to reach a consensus. The consensus process involved each reviewer presenting his or her observations and results, followed by discussion and debate. A third reviewer was available in cases of persistent disagreement or uncertainty. In the first step, all domains addressed by the quality items were identified. The first reviewer iteratively and progressively defined the domains as items were extracted from the included reviews. The second reviewer defined domains from a random list of all extracted quality items. Limited guidance was provided to the reviewers so that their assessments and definitions of domains would be independent. The reviewers agreed on a final set of domains that adequately and completely defined all of the extracted items. In the second step, reviewers independently grouped each extracted item into the appropriate domains. Reviewers considered each extracted item by asking, What is each particular quality item addressing? or What are the review's authors getting at with the particular quality assessment item?. Items were grouped into the domain or domains that best represented the concepts being addressed. For example, the extracted items at least 80% of the group originally identified was located for follow-up and follow-up was sufficiently complete or doesn't jeopardize validity were each independently classified by both reviewers as assessing the domain completeness of follow-up adequate, whereas the extracted item quantification and description of all subjects lost to follow-up was classified as assessing the domain completeness of follow-up described. In the third step, we identified the domains necessary to assess potential biases. Each reviewer considered the ability of the identified domains to adequately address, at least in part, 1 of the following 6 potential biases: 1) study participation, 2) study attrition, 3) prognostic factor measurement, 4) confounding measurement and account, 5) outcome measurement, and 6) analysis. Domains were considered to adequately address part of the framework if information garnered from that domain would inform the assessment of potential bias. For example, both reviewers judged that the identified domain study population represents source population or population of interest assessed potential bias in a prognosis study, whereas the domain research question definition did not, although the latter is an important consideration in assessing the inclusion of studies in a systematic review. Finally, on the basis of our previous ratings, we looked at whether each review included items from the domains necessary to assess the 6 potential biases. We calculated the frequency of systematic reviews by assessing each potential bias and the number of reviews that adequately assessed bias overall. From this systematic synthesis, we developed recommendations for improving quality appraisal in future systematic reviews of prognosis studies. We used Microsoft Access and Excel 2002 (Microsoft Corp., Redmond, Washington) for data management and SAS for Windows, version 9.1 (SAS Institute, Inc., Cary, North Carolina) for descriptive statistics. Role of the Funding Sources The funding sources, the Canadian Institutes of Health Research, the Canadian Chiropractic Research Foundation, the Ontario Chiropractic Association, and the Ontario Ministry of Health and Long Term Care, did not have a role in the collection, analysis, or interpretation of the data or in the decision to submit the manuscript for publication. Results We identified 1384 potentially relevant articles. Figure 1 shows a flow chart of studies that were included and excluded. Figure 2 shows the number of reviews identified by year of publication. We excluded 131 systematic reviews of prognosis studies that did not seem to include any quality assessment of the included studies; this represented 44% of prognosis reviews. We included 163 reviews of prognosis studies in our analysis (13-175). The most common topics were cancer (15%), musculoskeletal disorders and rheumatology (13%), cardiovascular (10%), neurology (10%), and obstetrics (10%). Other reviews included a wide range of health and health care topics. Sixty-three percent of the reviews investigated the association between a specific prognostic factor and a particular outcome; the remainder investigated multiple prognostic factors or models. The number of primary studies included in each systematic review ranged from 3 to 167 (median, 18 [interquartile range, 12 to 31]). A complete description of the included reviews is available from the authors on request. Figure 1. Flow diagram of inclusion and exclusion criteria of systematic reviews. Figure 2. Number of systematic reviews of prognosis studies identified over time. Quality Items One hundred fifty-three reviews provided adequate detail to allow extraction of quality items. Eight hundred eighty-two distinct quality items were extracted from the reviews. Most reviews developed their own set of quality items, with only a few applying criteria from previous reviews. Most quality items extracted