Semi-Automated Sensitivity Analysis to Assess Systematic Errors in Observational Data

Background Published epidemiologic research usually provides a quantitative assessment of random error for effect estimates, but no quantitative assessment of systematic error. Sensitivity analysis can provide such an assessment. Methods We describe a method to reconstruct epidemiologic data, accounting for biases, and to display the results of repeated reconstructions as an assessment of error. We illustrate with a study of the effect of less-than-definitive therapy on breast cancer mortality. Results We developed SAS code to reconstruct the data that would have been observed had a set of systematic errors been absent, and to convey the results. After 4,000 reconstructions of the example data, we obtained a median estimate of relative hazard equal to 1.5 with a 95% simulation interval of 0.8-2.8. The relative hazard obtained by conventional analysis equaled 2.0, with a 95% confidence interval of 1.2-3.4. Conclusions Our method of sensitivity analysis can be used to quantify the systematic error for an estimate of effect and to describe that error in figures, tables, or text. In the example, the sources of error biased the conventional relative hazard away from the null, and that error was not accurately communicated by the conventional confidence interval.

[1]  S. Schneeweiss,et al.  Causation of Bias: The Episcope , 2001, Epidemiology.

[2]  C Poole,et al.  Confidence intervals exclude nothing. , 1987, American journal of public health.

[3]  C R Weinberg.  It's time to rehabilitate the P-value. , 2001, Epidemiology.

[4]  S Greenland,et al.  Probability Logic and Probabilistic Induction , 1998, Epidemiology.

[5]  S Greenland,et al.  The effect of misclassification in the presence of covariates. , 1980, American journal of epidemiology.

[6]  M..  THE ROLE OF MODEL SELECTION IN CAUSAL INFERENCE FROM NONEXPERIMENTAL DATA , .

[7]  W. Eyler.  Uniform requirements for manuscripts submitted to biomedical journals. , 1980, Canadian Medical Association journal.

[8]  C. Baynes,et al.  A Hospital‐Based Case‐Control Study of Stillbirths and Environmental Exposure to Arsenic Using an Atmospheric Dispersion Model Linked to a Geographical Information System , 1998, Epidemiology.

[9]  J M Robins,et al.  The role of model selection in causal inference from nonexperimental data. , 1986, American journal of epidemiology.

[10]  V Mor,et al.  Age as a predictor of diagnostic and initial treatment intensity in newly diagnosed breast cancer patients. , 1989, Journal of gerontology.

[11]  W. Thompson,et al.  On the comparison of effects. , 1987, American journal of public health.

[12]  S. Norell,et al.  Workbook of Epidemiology , 1995 .

[13]  M. Levine,et al.  The Steering Committee on Clinical Practice Guidelines for the Care and Treatment of Breast Cancer. , 1998, CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne.

[14]  S Greenland,et al.  Randomization, Statistics, and Causal Inference , 1990, Epidemiology.

[15]  J M Robins,et al.  Identifiability, exchangeability, and epidemiological confounding. , 1986, International journal of epidemiology.

[16]  K J Rothman,et al.  That confounded P-value. , 1998, Epidemiology.

[17]  M. Szklo,et al.  Epidemiology: Beyond the Basics , 1999 .

[18]  Vincent Mor,et al.  The effect of less than definitive care on breast carcinoma recurrence and mortality , 2000, Cancer.

[19]  C Rouquette,et al.  [Epidemiologic research]. , 1970, Bulletin de l'Institut national de la sante et de la recherche medicale.

[20]  A. DeMaris.  Introduction to Regression Modeling , 2005 .

[21]  Mastectomy or lumpectomy? The choice of operation for clinical stages I and II breast cancer. The Steering Committee on Clinical Practice Guidelines for the Care and Treatment of Breast Cancer. Canadian Association of Radiation Oncologists. , 1998, CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne.

[22]  Uniform requirements for manuscripts submitted to biomedical journals. First edition, January 1, 1980. International Steering Committee. , 1979, The American review of respiratory disease.

[23]  D A Savitz,et al.  The effects of sensitivity and specificity of case selection on validity, sample size, precision, and power in hospital-based case-control studies. , 1990, American journal of epidemiology.

[24]  Sander Greenland,et al.  Theoretical Epidemiology: Principles of Occurrence Research in Medicine , 1986 .

[25]  J K McLaughlin,et al.  Selection of controls in case-control studies. I. Principles. , 1992, American journal of epidemiology.

[26]  B W Shaw,et al.  The value of P. , 1995, Liver transplantation and surgery : official publication of the American Association for the Study of Liver Diseases and the International Liver Transplantation Society.

[27]  Donald B. Rubin,et al.  Bayesian Inference for Causal Effects: The Role of Randomization , 1978 .

[28]  J. Buring,et al.  Epidemiology in Medicine , 1987 .

[29]  T. Lash,et al.  A Sensitivity Analysis to Separate Bias Due to Confounding from Bias Due to Predicting Misclassification by a Variable That Does Both , 2000, Epidemiology.

[30]  W. Thompson,et al.  Statistical criteria in the interpretation of epidemiologic data. , 1987, American journal of public health.

[31]  T. Lash,et al.  Re: Insulin-like growth factor 1 and prostate cancer risk: a population-based, case-control study. , 1998, Journal of the National Cancer Institute.

[32]  Kenneth J Rothman,et al.  Is flutamide effective in patients with bilateral orchiectomy? , 1999, The Lancet.

[33]  C Poole,et al.  Low P-Values or Narrow Confidence Intervals: Which Are More Durable? , 2001, Epidemiology.

[34]  C Poole,et al.  Beyond the confidence interval. , 1987, American journal of public health.