Issues in the selection of a summary statistic for meta‐analysis of clinical trials with binary outcomes

Meta-analysis of binary data involves the computation of a weighted average of summary statistics calculated for each trial. The selection of the appropriate summary statistic is a subject of debate due to conflicts in the relative importance of mathematical properties and the ability to intuitively interpret results. This paper explores the process of identifying a summary statistic most likely to be consistent across trials when there is variation in control group event rates. Four summary statistics are considered: odds ratios (OR); risk differences (RD) and risk ratios of beneficial (RR(B)); and harmful outcomes (RR(H)). Each summary statistic corresponds to a different pattern of predicted absolute benefit of treatment with variation in baseline risk, the greatest difference in patterns of prediction being between RR(B) and RR(H). Selection of a summary statistic solely based on identification of the best-fitting model by comparing tests of heterogeneity is problematic, principally due to low numbers of trials. It is proposed that choice of a summary statistic should be guided by both empirical evidence and clinically informed debate as to which model is likely to be closest to the expected pattern of treatment benefit across baseline risks. Empirical investigations comparing the four summary statistics on a sample of 551 systematic reviews provide evidence that the RR and OR models are on average more consistent than RD, there being no difference on average between RR and OR. From a second sample of 114 meta-analyses evidence indicates that for interventions aimed at preventing an undesirable event, greatest absolute benefits are observed in trials with the highest baseline event rates, corresponding to the model of constant RR(H). The appropriate selection for a particular meta-analysis may depend on understanding reasons for variation in control group event rates; in some situations uncertainty about the choice of summary statistic will remain.

[1]  S G Thompson,et al.  Investigating underlying risk as a source of heterogeneity in meta-analysis. , 1997, Statistics in medicine.

[2]  Douglas G Altman,et al.  Odds ratios should be avoided when events are common , 1998, BMJ.

[3]  F. Song,et al.  Cholesterol lowering and mortality: the importance of considering initial level of risk. , 1993, BMJ.

[4]  C H Schmid,et al.  An empirical study of the effect of the control rate as a predictor of treatment efficacy in meta-analysis of clinical trials. , 1998, Statistics in medicine.

[5]  A G Marson,et al.  Lamotrigine add-on for drug-resistant partial epilepsy. , 2001, The Cochrane database of systematic reviews.

[6]  Brendan Delaney,et al.  Systematic review and economic evaluation of Helicobacter pylori eradication treatment for non-ulcer dyspepsia , 2000, BMJ : British Medical Journal.

[7]  Allen S. Mandel.  Comment … , 1978, British heart journal.

[8]  D. Sackett,et al.  The number needed to treat: a clinically useful measure of treatment effect , 1995, BMJ.

[9]  D L Sackett,et al.  An assessment of clinically useful measures of the consequences of treatment. , 1988, The New England journal of medicine.

[10]  Douglas G Altman,et al.  The relation between treatment benefit and underlying risk in meta-analysis , 1996, BMJ.

[11]  N. Breslow,et al.  Statistical methods in cancer research: volume 1- The analysis of case-control studies , 1980 .

[12]  S D Walter,et al.  Choice of effect measure for epidemiological data. , 2000, Journal of clinical epidemiology.

[13]  I Olkin,et al.  Heterogeneity and statistical significance in meta-analysis: an empirical study of 125 meta-analyses. , 2000, Statistics in medicine.

[14]  J. Lubsen,et al.  Large trials with simple protocols: indications and contraindications. , 1989, Controlled clinical trials.

[15]  R. Peto,et al.  Beta blockade during and after myocardial infarction: an overview of the randomized trials. , 1985, Progress in cardiovascular diseases.

[16]  L. Laine,et al.  Therapy for Helicobacter pylori in Patients with Nonulcer Dyspepsia , 2001, Annals of Internal Medicine.

[17]  W. Haenszel,et al.  Statistical aspects of the analysis of data from retrospective studies of disease. , 1959, Journal of the National Cancer Institute.

[18]  J M Robins,et al.  Confounding and misclassification. , 1985, American journal of epidemiology.

[19]  S. Senn,et al.  IMPORTANCE OF TRENDS IN THE INTERPRETATION OF AN OVERALL ODDS RATIO IN THEMETA-ANALYSIS OF CLINICAL TRIALS. AUTHOR'S REPLY , 1994 .

[20]  Eliseo Guallar,et al.  A graphical display useful for meta-analysis , 1997 .

[21]  A. Feinstein.  Indexes of contrast and quantitative significance for comparisons of two groups. , 1999, Statistics in medicine.

[22]  Lisa M. Schwartz,et al.  Misunderstandings about the effects of race and sex on physicians' referrals for cardiac catheterization. , 1999, The New England journal of medicine.

[23]  H Kragt,et al.  Importance of trends in the interpretation of an overall odds ratio in the meta-analysis of clinical trials. , 1992, Statistics in medicine.

[24]  M. Patel,et al.  Polysaccharide vaccines for preventing serogroup A meningococcal meningitis. , 2005, The Cochrane database of systematic reviews.

[25]  Les M Irwig,et al.  An evidence based approach to individualising treatment , 1995, BMJ.

[26]  D. Sackett,et al.  70 Choosing an appropriate dichotomous effect measure for meta-analysis: Empirical evidence of the appropriateness of the odds ratio and relative risk , 1997 .

[27]  Jenny Field,et al.  General practitioners' perceptions of the route to evidence based medicine: a questionnaire survey , 1998, BMJ.

[28]  P. Wiffen,et al.  Single dose oral aspirin for acute pain. , 1999, The Cochrane database of systematic reviews.

[29]  H. Davies,et al.  When can odds ratios mislead? , 1998, BMJ.

[30]  M. Bracken,et al.  Clinically useful measures of effect in binary analyses of randomized trials. , 1994, Journal of clinical epidemiology.

[31]  J M Robins,et al.  Estimation of a common effect parameter from sparse follow-up data. , 1985, Biometrics.

[32]  J. Deeks.  When can odds ratios mislead? , 1998 .

[33]  M. Egger,et al.  Who benefits from medical interventions? , 1994, BMJ.

[34]  B. Everitt,et al.  Statistical methods for rates and proportions , 1973 .

[35]  S D Walter,et al.  Variation in baseline risk as an explanation of heterogeneity in meta-analysis. , 1997, Statistics in medicine.

[36]  Sj Senn,et al.  Odds ratios revisited , 1998 .

[37]  David R. Cox.  The analysis of binary data , 1970 .

[38]  M. McIntosh,et al.  The population risk as an explanatory variable in research synthesis of clinical trials. , 1996, Statistics in medicine.

[39]  Jonathan J. Deeks,et al.  Down with odds ratios! , 1996, Evidence Based Medicine.

[40]  E. Genton,et al.  Observations in anticoagulant and thrombolytic therapy in pulmonary embolism. , 1975, Progress in cardiovascular diseases.

[41]  S. Thompson,et al.  Quantifying heterogeneity in a meta‐analysis , 2002, Statistics in medicine.

[42]  Jonathan J Deeks,et al.  metan-an alternative meta-analysis command , 1999 .