Challenges in Systematic Reviews of Complementary and Alternative Medicine Topics

The popularity of complementary and alternative medicine (CAM) in the United States continues to grow (1). The Agency for Healthcare Research and Quality (AHRQ) has devoted a substantial proportion of the Evidence-based Practice Center (EPC) program to reports on CAM. As of 2004, 19 of the more than 100 evidence reports concerned CAM interventions. The reports spanned a wide scope of topics, including botanicals and herbal supplements (such as garlic and ephedra), traditional medicine (such as acupuncture and ayurvedic medicine), and vitamins (such as supplementation). In preparing CAM reports, EPC reviewers faced several challenges. In this paper, we discuss approaches used to address the following 3 issues: 1) identifying evidence about CAM interventions, 2) assessing the quality of individual studies, and 3) addressing rare serious adverse events. We then suggest recommendations in each area for future reviews of CAM interventions. Challenge: Identifying Evidence about CAM Literature The many biases in the publication and indexing of CAM research pose a challenge for locating literature. Publication bias refers to the tendency of investigators, reviewers, and editors to submit or accept manuscripts on the basis of the strength or direction of the findings. While publication bias is a concern in conventional medical research (2-4), in CAM research the issue is particularly complicated. Most studies published in leading CAM journals have positive results (5). Some countries, such as China, Japan, Russia, and Taiwan, publish more studies with positive results than studies with negative results; this imbalance may reflect publication bias (6). Language-related publication bias also exists in CAM research, as it does for conventional medicine research (7). Negative CAM findings are more likely to be published in mainstream medical journals (that is, English-language journals), while positive CAM findings are more likely to be published in CAM journals (which tend to be nonEnglish-language journals) (8-10). Thus, all else being equal, the exclusion of nonEnglish-language reports may result in a lower estimate of intervention effect for CAM interventions than would be seen if these reports were included (10). The direction of bias, however, may depend on the CAM topic. Studies about some CAM therapies may initially be published only in nonEnglish-language journals (8). On the other hand, in pediatric CAM, some evidence suggests that randomized, controlled trials (RCTs) tend to be published in English, and often in mainstream medical journals (9). Another bias that hinders access is the incomplete or improper indexing of CAM journals and articles by mainstream databases such as MEDLINE (11, 12). For example, MEDLINE indexed only 10% of CAM journals identified worldwide by the National Center for Complementary and Alternative Medicine and the National Library of Medicine (12), while approximately 35% of all biomedical journals published worldwide are indexed in MEDLINE (13). Inconsistent use of keywords, descriptors, and subject headings, along with differing indexing procedures across databases, may also pose a challenge in locating CAM literature (14). The inconsistency in the terminology may partially explain why a MEDLINE search of the term alternative medicine does not capture all studies relevant to the CAM field (12). We examined the 19 evidence reports and 2 technology assessments related to CAM for sources and methods used to locate CAM literature (Appendix Table). For almost all reports, investigators searched specialized CAM databases in addition to mainstream databases. At least 10 databases specialize in CAM: Allied and Complementary Medicine Database (AMED); Cochrane Complementary Medicine Trials Registry; Alt HealthWatch; HerbMed; Manual, Alternative and Natural Therapy Index System (MANTIS); Natural Products Alert (NAPRALERT); International Bibliographic Information on Dietary Supplements (IBIDS); Hom-Inform; The Arthritis and Complementary Medicine Database (ARCAM); and Complementary and Alternative Medicine and Pain Database (CAMPAIN). Many can be accessed without a subscription. The search strategy used for most reports did not involve English-language restrictions. For almost all reports, investigators included an extensive list of keywords to identify articles that may not be indexed by standard subject headings, or for which terminology was inconsistent (14). Only a few investigators used hand searching (15). The searching of grey literature is another method that slightly more than half of the investigators used to address publication bias. Grey literature was searched in about half the reports. For most reports, investigators specified that they had reviewed reference lists to identify articles improperly indexed by mainstream databases. However, investigators should be aware of the possibility of citation bias, a phenomenon whereby studies supportive of a beneficial effect are cited more frequently (16). There is evidence of this bias in mainstream medical literature (16), and it could be relevant to CAM. Similarly, for most reports, investigators specified that they had communicated with authors or experts for additional data and relevant studies. However, investigators should be aware of the possibility of bias in the provision of information, such that studies with positive results may be more likely to be communicated to investigators than are studies with negative results (7). Challenge: Assessing and Summarizing Quality Complementary and alternative medicine presents special challenges in the design and execution of studies, with respect to both internal validity and generalizability. These problems relate to the tension between specifying the intervention sufficiently that others can apply it and the desire to study CAM therapy as it is applied in CAM practices; they also concern the difficulties in controlling expectation bias (the systematic effect on the results of the participants' belief that a certain therapy will help them). Most CAM interventions are investigated only after they are so widespread that they can no longer be ignored, and by that time, the CAM practices are highly diversified in practices, personal experiences, biases, and expectations. For some CAM therapies, specifying the intervention sufficiently is not a great problem, at least in theory. The use of herbs or dietary supplements such as St. John's wort, antioxidant supplements, glucosamine, and saw palmetto can be specified in a manner analogous to that used in drug trials, with a standard concentration or formula given regularly. (However, there is concern about the variation in formulation and bioactivity of some supplements from lot to lot, and this variation may be of even greater concern for botanicals in which the active ingredient is not known.) For many other CAM therapies, however, the conceptual basis for the therapy requires an interaction between the practitioner and patient that modifies the therapy to the individual. Traditional Chinese acupuncture, spinal manipulation by chiropractors, and ayurvedic medicine require individualization of treatment based on an examination and understanding of the patient's condition using concepts that do not have an analogue in western allopathic medicine. Consequently, CAM advocates have criticized randomized clinical trials that reported no effect for not having allowed the necessary tailoring of the intervention. For this reason, pragmatic trials are frequently advocated for studying CAM. In a pragmatic trial, patients are assigned to a CAM practitioner rather than a tightly specified CAM therapy. The CAM practitioners can provide their treatments in their usual fashion, individualizing the therapy for each particular patient. While this strategy allows the CAM practice to occur in its traditional fashion, it makes blinding or otherwise controlling expectation bias very difficult. Furthermore, while in one way individualizing the therapy increases generalizability, it also increases the sensitivity of the results to the skill of the practitioners. Since the intervention relies on practitioner expertise in understanding the patient and delivering the therapy, the study results are more difficult to apply to other practitioners. Thus, pragmatic trials should discuss the training and experience of the CAM practitioner. Large pragmatic trials that include many practitioners and that compare a CAM therapy with a credible control or alternative therapy would be particularly useful in assessing CAM. The traditional way to manage expectation bias in drug trials is to use a matched placebo and a double-blind design. The magnitude of a potential placebo response, or even its existence at all, is controversial (17-20). However, there is agreement that the placebo response is particularly important in studies that use subjective patient measures of outcome, especially pain. Because chronic painful conditions account for a large proportion of the reasons patients seek CAM therapies, the use of placebo controls is especially important. Even medical procedures such as knee surgery have been shown to be amenable to studies with a sham control (21), so the issue of placebo and shams spans the gamut of CAM therapies. If CAM treatments, such as herbs and dietary supplements, are amenable to a matched-placebo, double-blind design, anything less is a limitation. For many other CAM therapies, however, placebos and blinding are a much greater challenge. The use of sham procedures, rather than placebos, is possible when the CAM therapy being evaluated is well-specified. One mechanism for assessing the success of the blinding ability of a sham therapy is to survey patients after receipt of the therapy and ask them to guess whether they received active or sham therapy. Similar responses in patients receiving active and sham therapy are good evidence that the sham was successful in co