Assessing sample representativeness in randomized controlled trials: application to the National Institute of Drug Abuse Clinical Trials Network.

AIMS To compare the characteristics of individuals participating in randomized controlled trials (RCTs) of treatments of substance use disorder (SUD) with individuals receiving treatment in usual care settings, and to provide a summary quantitative measure of differences between characteristics of these two groups of individuals using propensity score methods. Design Analyses using data from RCT samples from the National Institute of Drug Abuse Clinical Trials Network (CTN) and target populations of patients drawn from the Treatment Episodes Data Set-Admissions (TEDS-A). Settings Multiple clinical trial sites and nation-wide usual SUD treatment settings in the United States. PARTICIPANTS A total of 3592 individuals from 10 CTN samples and 1 602 226 individuals selected from TEDS-A between 2001 and 2009. Measurements The propensity scores for enrolling in the RCTs were computed based on the following nine observable characteristics: sex, race/ethnicity, age, education, employment status, marital status, admission to treatment through criminal justice, intravenous drug use and the number of prior treatments. Findings The proportion of those with ≥ 12 years of education and the proportion of those who had full-time jobs were significantly higher among RCT samples than among target populations (in seven and nine trials, respectively, at P < 0.001). The pooled difference in the mean propensity scores between the RCTs and the target population was 1.54 standard deviations and was statistically significant at P < 0.001. CONCLUSIONS In the United States, individuals recruited into randomized controlled trials of substance use disorder treatments appear to be very different from individuals receiving treatment in usual care settings. Notably, RCT participants tend to have more years of education and a greater likelihood of full-time work compared with people receiving care in usual care settings.

[1]  Philip S. Wang,et al.  Improving mental health treatments through comparative effectiveness research. , 2009, Health affairs.

[2]  T. Killeen,et al.  Effect of prize-based incentives on outcomes in stimulant abusers in outpatient psychosocial treatment programs: a national drug abuse treatment clinical trials network study. , 2005, Archives of general psychiatry.

[3]  Ali Anaissi,et al.  A balanced iterative random forest for gene selection from microarray data , 2013, BMC Bioinformatics.

[4]  Douglas G. Altman,et al.  Metan: Fixed- and Random-Effects Meta-Analysis , 2008 .

[5]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[6]  Dawn E. Sugarman,et al.  The relationship between educational attainment and relapse among alcohol-dependent men and women: a prospective study. , 2003, Alcoholism, clinical and experimental research.

[7]  Jeffrey J. Annon,et al.  A multi-center randomized trial of buprenorphine-naloxone versus clonidine for opioid detoxification: findings from the National Institute on Drug Abuse Clinical Trials Network. , 2005, Addiction.

[8]  Elizabeth A Stuart,et al.  Matching methods for causal inference: A review and a look forward. , 2010, Statistical science : a review journal of the Institute of Mathematical Statistics.

[9]  J. Rabkin,et al.  Predictors of Employment of Men With HIV/AIDS: A Longitudinal Study , 2004, Psychosomatic medicine.

[10]  Sandra E. Larios,et al.  Increasing ethnic minority participation in substance abuse clinical trials: lessons learned in the National Institute on Drug Abuse's Clinical Trials Network. , 2011, Cultural diversity & ethnic minority psychology.

[11]  The influence of patients' preference/attitude towards psychotherapy and antidepressant medication on the treatment of major depressive disorder. , 2014, Journal of behavior therapy and experimental psychiatry.

[12]  C. G. Hudson,et al.  Socioeconomic status and mental illness: tests of the social causation and selection hypotheses. , 2005, The American journal of orthopsychiatry.

[13]  P. Austin,et al.  Reader's guide to critical appraisal of cohort studies: 2. Assessing potential for confounding , 2005, BMJ : British Medical Journal.

[14]  J. Last,et al.  Making the Dictionary of Epidemiology. , 1996, International journal of epidemiology.

[15]  Roger A. Sugden,et al.  Multiple Imputation for Nonresponse in Surveys , 1988 .

[16]  Mark Olfson,et al.  Generalizability of clinical trials for cannabis dependence to community samples. , 2008, Drug and alcohol dependence.

[17]  S. Lipsitz,et al.  Challenges and recommendations for blinding in behavioral interventions illustrated using a case study of a behavioral intervention to lower blood pressure. , 2010, Patient education and counseling.

[18]  P. Ferrinho,et al.  Prognostic factors during outpatient treatment for alcohol dependence: cohort study with 6 months of treatment follow-up. , 2012, Alcohol and alcoholism.

[19]  G. Woody,et al.  Motivational interviewing to improve treatment engagement and outcome in individuals seeking treatment for substance abuse: a multisite effectiveness study. , 2006, Drug and alcohol dependence.

[20]  R. Groves,et al.  Survey Errors and Survey Costs. , 1991 .

[21]  M. Kenward,et al.  Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls , 2009, BMJ : British Medical Journal.

[22]  P. Lachenbruch.  Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[23]  Gary King,et al.  Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference , 2007, Political Analysis.

[24]  Catherine P. Bradshaw,et al.  The use of propensity scores to assess the generalizability of results from randomized trials , 2011, Journal of the Royal Statistical Society. Series A,.

[25]  G. Tutz,et al.  An introduction to recursive partitioning: rationale, application, and characteristics of classification and regression trees, bagging, and random forests. , 2009, Psychological methods.

[26]  A. Stotts,et al.  Determining predictors of attrition in an outpatient substance abuse program , 2002, The American journal of drug and alcohol abuse.

[27]  R. Putnam.  Tuning In, Tuning Out: The Strange Disappearance of Social Capital in America , 1995, PS: Political Science &amp; Politics.

[28]  Elizabeth L. Ogburn,et al.  Generalizability of clinical trial results for major depression to community samples: results from the National Epidemiologic Survey on Alcohol and Related Conditions. , 2008, The Journal of clinical psychiatry.

[29]  Jeffrey J. Annon,et al.  Buprenorphine tapering schedule and illicit opioid use. , 2009, Addiction.

[30]  Andy Liaw,et al.  Classification and Regression by randomForest , 2007 .

[31]  N. Petry,et al.  Effects of lower-cost incentives on stimulant abstinence in methadone maintenance treatment: a National Drug Abuse Treatment Clinical Trials Network study. , 2006, Archives of general psychiatry.

[32]  Elizabeth A Stuart,et al.  Improving propensity score weighting using machine learning , 2010, Statistics in medicine.

[33]  Hee-Je Bak,et al.  Education and Public Attitudes toward Science: Implications for the "Deficit Model" of Education and Support for Science and Technology , 2001 .

[34]  D. Rubin.  Matched Sampling for Causal Effects: The Use of Matched Sampling and Regression Adjustment to Remove Bias in Observational Studies , 1973 .

[35]  Frank Sullivan,et al.  Strategies to improve recruitment to randomised controlled trials. , 2010, The Cochrane database of systematic reviews.

[36]  Jennifer Hill,et al.  Assessing Methods for Generalizing Experimental Impact Estimates to Target Populations , 2016, Journal of research on educational effectiveness.

[37]  V. Harder,et al.  The increase in the association of education and cocaine use over the 1980s and 1990s: evidence for a 'historical period' effect. , 2005, Drug and alcohol dependence.

[38]  K. Humphreys,et al.  Use of exclusion criteria in selecting research subjects and its effect on the generalizability of alcohol treatment outcome studies. , 2000, The American journal of psychiatry.

[39]  F. Smit,et al.  Risk factors for 12-month comorbidity of mood, anxiety, and substance use disorders: findings from the Netherlands Mental Health Survey and Incidence Study. , 2002, The American journal of psychiatry.

[40]  G. Woody,et al.  Site matters: multisite randomized trial of motivational enhancement therapy in community drug abuse clinics. , 2007, Journal of consulting and clinical psychology.

[41]  Sarah J. Erickson,et al.  Motivational enhancement therapy to improve treatment utilization and outcome in pregnant substance users. , 2008, Journal of substance abuse treatment.

[42]  C. Latkin,et al.  Neighborhood socioeconomic status, personal network attributes, and use of heroin and cocaine. , 2007, American journal of preventive medicine.

[43]  W. G. Cochran,et al.  Controlling Bias in Observational Studies: A Review. , 1974 .

[44]  A Oakley,et al.  Assessment of generalisability in trials of health interventions: suggested framework and systematic review , 2006, BMJ : British Medical Journal.

[45]  Chao Chen,et al.  Using Random Forest to Learn Imbalanced Data , 2004 .

[46]  D. Altman,et al.  Measuring inconsistency in meta-analyses , 2003, BMJ : British Medical Journal.

[47]  Alex H. S. Harris,et al.  Influence of subject eligibility criteria on compliance with National Institutes of Health guidelines for inclusion of women, minorities, and children in treatment research. , 2007, Alcoholism, clinical and experimental research.

[48]  K. Dugosh,et al.  Extended vs. short‐term buprenorphinenaloxone for treatment of opioid addicted youth: A randomized trial , 2008, JAMA.

[49]  M Egger,et al.  The causes and effects of socio-demographic exclusions from clinical trials. , 2005, Health technology assessment.

[50]  I. Anderson,et al.  Area-level socioeconomic status in relation to outcomes in γ-hydroxybutyrate intoxication , 2009, Clinical toxicology.

[51]  S. Sonne,et al.  Adjunctive counseling during brief and extended buprenorphine-naloxone treatment for prescription opioid dependence: a 2-phase randomized controlled trial. , 2011, Archives of general psychiatry.