Evaluating Disease Management Program Effectiveness

Disease management (DM) program evaluations are somewhat limited in scope because of typically small sample sizes comprising important subsets of the treated population. Identifying subsets of the data that have differing results from the aggregate of the whole program can lend insight into where, when, and how the program achieves its results. Additionally, there is a very limited set of classical tools available for the smaller sample sizes typically encountered in DM. Without readily available standard error and confidence interval (CI) calculations, the analyst may be fooled by specious details.A method called the ‘bootstrap’ is introduced as a suitable technique for allowing DM program evaluators to use a broader array of quantities of interest and to extend inferences to the population based on results achieved in the program. The bootstrap uses the power of modern computers to generate many random samples from a given data set, allowing the use of repeated samples’ statistic (e.g. mean, proportion, and median). Using a congestive heart failure (CHF) program as an example, the bootstrap technique is used to extend a DM program evaluation beyond questions addressed using classical statistical inference: (i) how much of a median cost decrease can be expected as a result of the program?; (ii) did the program impact the highest and lowest costing members equally; and (iii) how much of a decrease in the proportion of patients experiencing a hospitalization can be expected as a result of the program?The potential advantages of the bootstrap technique in DM program evaluation were clearly illustrated using this small CHF program example. A more robust understanding of program impact is possible when more tools and methods are available to the evaluator. This is particularly the case in DM, which is inherently biased in case-mix (e.g. strive to enroll sickest first), often has skewed distributions or outliers, and may suffer from small sample sizes.The bootstrap technique creates distributions that allow for a more accurate method of drawing statistical inferences of a population. Moreover, since classical statistical inference techniques were designed specifically for parametric statistics (i.e. assuming a normal distribution), the bootstrap can be used for measures that have no convenient statistical formulae. Additionally, CIs can be defined around this statistic, making it a viable option for evaluating DM program effectiveness.

[1]  S. T. Buckland,et al.  An Introduction to the Bootstrap. , 1994 .

[2]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[3]  Ariel Linden,et al.  Disease management interventions: what's in the black box? , 2004, Disease management : DM.

[4]  N. Draper,et al.  Applied Regression Analysis , 1966 .

[5]  D G Altman,et al.  Time to event (survival) data , 1998, BMJ.

[6]  Using an empirical method for establishing clinical outcome targets in disease management programs. , 2004, Disease management : DM.

[7]  Charles S. Reichardt,et al.  Justifying the use and increasing the power of a t test for a randomized experiment with a convenience sample. , 1999 .

[8]  David R. Cox,et al.  Regression models and life tables (with discussion , 1972 .

[9]  J. Simpson,et al.  Secondary prevention clinics for coronary heart disease: four year follow up of a randomised controlled trial in primary care , 2003, BMJ : British Medical Journal.

[10]  Evaluating Disease Management Program Effectiveness Adjusting for Enrollment (Tenure) and Seasonality , 2004 .

[11]  C. Lunneborg.  Data Analysis by Resampling: Concepts and Applications , 1999 .

[12]  G. A. Barnard,et al.  Student: A Statistical Biography of William Sealy Gosset , 1990 .

[13]  Rajiv Vaidyanathan,et al.  Dictionary of statistics and methodology : #a #nontechnical guide for the social sciences , 1994 .

[14]  J. Kalbfleisch,et al.  The Statistical Analysis of Failure Time Data , 1980 .

[15]  J. Concato,et al.  The Risk of Determining Risk with Multivariable Models , 1993, Annals of Internal Medicine.

[16]  Anthony C. Davison,et al.  Bootstrap Methods and Their Application , 1998 .

[17]  Frederick Mosteller,et al.  Data Analysis and Regression , 1978 .

[18]  Ariel Linden,et al.  Using Propensity Scores to Construct Comparable Control Groups for Disease Management Program Evaluation , 2005 .

[19]  J. Adams,et al.  An assessment of the total population approach for evaluating disease management program effectiveness. , 2003, Disease management : DM.

[20]  E. Kaplan,et al.  Nonparametric Estimation from Incomplete Observations , 1958 .

[21]  J. Adams,et al.  Strengthening the case for disease management effectiveness: un-hiding the hidden bias. , 2006, Journal of evaluation in clinical practice.

[22]  M. Pike,et al.  Design and analysis of randomized clinical trials requiring prolonged observation of each patient. II. analysis and examples. , 1977, British Journal of Cancer.

[23]  D. Campbell,et al.  EXPERIMENTAL AND QUASI-EXPERIMENT Al DESIGNS FOR RESEARCH , 2012 .

[24]  Michael R. Chernick,et al.  Bootstrap Methods: A Practitioner's Guide , 1999 .

[25]  J. Adams,et al.  Evaluating disease management program effectiveness: an introduction to survival analysis. , 2004, Disease management : DM.

[26]  John Ludbrook,et al.  Why Permutation Tests are Superior to t and F Tests in Biomedical Research , 1998 .

[27]  Ariel Linden,et al.  Evaluating disease management program effectiveness: an introduction to time-series analysis. , 2003, Disease management : DM.