Evaluation of methods for interval estimation of model outputs, with application to survival models

When a published statistical model is also distributed as computer software, it will usually be desirable to present the outputs as interval, as well as point, estimates. The present paper compares three methods for approximate interval estimation about a model output, for use when the model form does not permit an exact interval estimate. The methods considered are first-order asymptotics, using second derivatives of the log-likelihood to estimate variance information; higher-order asymptotics based on the signed-root transformation; and the non-parametric bootstrap. The signed-root method is Bayesian, and uses an approximation for posterior moments that has not previously been tested in a real-world application. Use of the three methods is illustrated with reference to a software project arising in medical decision-making, the UKPDS Risk Engine. Intervals from the first-order and signed-root methods are near- identical, and typically 1% wider to 7% narrower than those from the non-parametric bootstrap. The asymptotic methods are markedly faster than the bootstrap method.

[1]  R. Holman,et al.  Life-expectancy projection by modelling and computer simulation (UKPDS 46). , 2000, Diabetes research and clinical practice.

[2]  Thomas J. DiCiccio,et al.  Approximations of marginal tail probabilities and inference for scalar parameters , 1990 .

[3]  R. Holman,et al.  The UKPDS risk engine: a model for the risk of coronary heart disease in Type II diabetes (UKPDS 56). , 2001, Clinical science.

[4]  Deborah Burr,et al.  A Comparison of Certain Bootstrap Confidence Intervals in the Cox Model , 1994 .

[5]  Uk-Prospective-Diabetes-Study-Group.  Intensive blood-glucose control with sulphonylureas or insulin compared with conventional treatment and risk of complications in patients with type 2 diabetes (UKPDS 33) , 1998, The Lancet.

[6]  T. DiCiccio,et al.  Approximations of marginal tail probabilities for a class of smooth functions with applications to Bayesian and conditional inference , 1991 .

[7]  C. Chatfield.  Model uncertainty, data mining and statistical inference , 1995 .

[8]  William H. Press,et al.  Book-Review - Numerical Recipes in Pascal - the Art of Scientific Computing , 1989 .

[9]  J Carpenter,et al.  Bootstrap confidence intervals: when, which, what? A practical guide for medical statisticians. , 2000, Statistics in medicine.

[10]  N. L. Johnson,et al.  Systems of frequency curves generated by methods of translation. , 1949, Biometrika.

[11]  I. D. Hill,et al.  Fitting Johnson Curves by Moments , 1976 .

[12]  J. Mckenney,et al.  Executive Summary of The Third Report of The National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, And Treatment of High Blood Cholesterol In Adults (Adult Treatment Panel III). , 2001, JAMA.

[13]  Ukpds,et al.  Intensive blood-glucose control with sulphonylureas or insulin compared with conventional treatment and risk of complications in patients with type 2 diabetes , 2002 .

[14]  R. Holman,et al.  Tight blood pressure control and risk of macrovascular and microvascular complications in type 2 diabetes: UKPDS 38. UK Prospective Diabetes Study Group. , 1998 .

[15]  R. Tibshirani.  Noninformative priors for one parameter of many , 1989 .

[16]  Bradley Efron,et al.  Censored Data and the Bootstrap , 1981 .

[17]  Joint British recommendations on prevention of coronary heart disease in clinical practice: summary , 2001, BMJ : British Medical Journal.

[18]  Bliss Ci,et al.  Fitting the rectangular hyperbola. , 1966 .

[19]  B. Efron.  Bootstrap Methods: Another Look at the Jackknife , 1979 .

[20]  B. Efron.  Better Bootstrap Confidence Intervals , 1987 .

[21]  Coverage Probability Bias , Objective Bayes and theLikelihood , 1999 .

[22]  A. Zbrozek,et al.  Model of Complications of NIDDM: II. Analysis of the health benefits and cost-effectiveness of treating NIDDM with the goal of normoglycemia , 1997, Diabetes Care.

[23]  David Draper,et al.  Assessment and Propagation of Model Uncertainty , 2011 .

[24]  C. Field,et al.  An accurate method for approximate conditional and Bayesian inference about linear regression models from censored data , 1991 .

[25]  Trevor J. Sweeting,et al.  Approximate Bayesian computation based on signed roots of log-density ratios (with discussion) , 1996 .

[26]  K Borch-Johnsen,et al.  A New Method for Chd Prediction and Prevention Based on Regional Risk Scores and Randomized Clinical Trials; PRECARD® and the Copenhagen Risk Score , 2001, Journal of cardiovascular risk.

[27]  C. I. Bliss,et al.  Fitting the rectangular hyperbola. , 1966, Biometrics.

[28]  O. E. Barndorff-Nielsen,et al.  Infereni on full or partial parameters based on the standardized signed log likelihood ratio , 1986 .

[29]  Ole E. Barndorff-Nielsen,et al.  Approximate Interval Probabilities , 1990 .

[30]  E. Scott.  Modelling Survival Data in Medical Research , 1995 .

[31]  R. Holman,et al.  Intensive blood-glucose control with sulphonylureas or insulin compared with conventional treatment and risk of complications in patients with type 2 diabetes (UKPDS 33). UK Prospective Diabetes Study (UKPDS) Group. , 1998 .

[32]  Trevor J. Sweeting,et al.  A framework for Bayesian and likelihood approximations in statistics , 1995 .

[33]  A Novel Formulation for Approximate Bayesian Computation Based on Signed Roots of Log-Density Ratios , 2005 .

[34]  Trevor J. Sweeting,et al.  Coverage probability bias, objective Bayes and the likelihood principle , 2001 .

[35]  T. Sweeting.  A Bayesian approach to approximate conditional inference , 1995 .

[36]  D. Collett.  Modelling survival data , 1994 .

[37]  Anthony C. Davison,et al.  Bootstrap Methods and Their Application , 1998 .

[38]  R. Holman,et al.  Erratum: The UKPDS risk engine: A model for the risk of coronary heart disease in type II diabetes (UKPDS 56) (Clinical Science (2001) 101 (671-679)) , 2002 .

[39]  F. A. Seiler,et al.  Numerical Recipes in C: The Art of Scientific Computing , 1989 .

[40]  L. Tierney,et al.  Accurate Approximations for Posterior Moments and Marginal Densities , 1986 .

[41]  D. Cox,et al.  Parameter Orthogonality and Approximate Conditional Inference , 1987 .

[42]  R. Stevens,et al.  UKPDS 60: Risk of Stroke in Type 2 Diabetes Estimated by the UK Prospective Diabetes Study Risk Engine , 2002, Stroke.

[43]  J B Brown.  Computer models of diabetes: almost ready for prime time. , 2000, Diabetes research and clinical practice.

[44]  Philip D. Harvey,et al.  Cost effectiveness analysis of improved blood pressure control in hypertensive patients with type 2 diabetes: UKPDS 40 , 1998, BMJ.