Editor—Olsen et al assessed a sample of Cochrane reviews from 1998 and highlighted some areas where improvement is possible.1 They found that 29% of reviews had major problems, including inappropriate methods and conclusions. As they say, improvement is still possible, but this figure nevertheless represents a major improvement on the quality of non-Cochrane reviews.
We have reviewed the methods of 480 systematic reviews on the database of abstracts of reviews of effectiveness (DARE) at the University of York.2,3 Methodological details of the reviews were coded and checked by two reviewers working independently. We found that only half (52%) of the reviews had systematically assessed the validity of the included studies; that most systematic reviews were unlikely to be comprehensive (they had searched either one or two databases); and that overall only a quarter (26%) of reviews met three key methodological criteria (relating to a thorough search, assessment of the validity of the included studies, and investigation of heterogeneity). Narrative reviews were less likely to meet all three criteria (20% v 30%, P=0.02) and more likely to be coded by raters as inconclusive.
Up to half of non-Cochrane reviews are thus potentially misleading. Against this, Olsen et al's estimate of 29% for Cochrane reviews compares favourably. Although more recent research syntheses are likely to be of higher quality, particularly if reviewers follow current guidelines,4,5 problems with the reliability of systematic reviews will probably remain. Since our study was conducted the criteria for including systematic reviews on the database of abstracts of reviews of effectiveness have been revised (from October 2000 onwards) to ensure that only reviews of potentially high methodological quality are included.
We would support Olsen et al's suggestion that users of any systematic review should assess its reliability. We would also recommend that for a critical assessment of the quality of non-Cochrane reviews users should first look at the database of abstracts of reviews of effectiveness.