Blending of Probability and Non-Probability Samples: Applications to a Survey of Military Caregivers

Probability samples are the preferred method for providing inferences that are generalizable to a larger population. However, when a small (or rare) subpopulation is the group of interest, this approach is unlikely to yield a sample size large enough to produce precise inferences. Non-probability (or convenience) sampling often provides the necessary sample size to yield efficient estimates, but selection bias may compromise the generalizability of results to the broader population. Motivating the exposition is a survey of military caregivers; our interest is focused on unpaid caregivers of wounded, ill, or injured servicemembers and veterans who served in the US armed forces following September 11, 2001. An extensive probability sampling effort yielded only 72 caregivers from this subpopulation. Therefore, we consider supplementing the probability sample with a convenience sample from the same subpopulation, and we develop novel methods of statistical weighting that may be used to combine (or blend) the samples. Our analyses show that the subpopulation of interest endures greater hardships than caregivers of veterans with earlier dates of service, and these conclusions are discernably stronger when blended samples with the proposed weighting schemes are used. We conclude with simulation studies that illustrate the efficacy of the proposed techniques, examine the bias-variance trade-off encountered when using inadequately blended data, and show that the gain in precision provided by the convenience sample is lower in circumstances where the outcome is strongly related to the auxiliary variables used for blending.

[1]  Sunghee Lee Propensity score adjustment as a weighting scheme for volunteer panel web surveys , 2006 .

[2]  M. Elliott,et al.  Use of a web­based convenience sample to supplement a probability sample , 2007 .

[3]  M. Frölich Non-Parametric Regression for Binary Dependent Variables , 2006 .

[4]  P. Kott Using Calibration Weighting to Adjust for Nonresponse and Coverage Errors , 2006 .

[5]  Sunghee Lee,et al.  Estimation for Volunteer Panel Web Surveys Using Propensity Score Adjustment and Calibration Adjustment , 2009 .

[6]  D. Horvitz,et al.  A Generalization of Sampling Without Replacement from a Finite Universe , 1952 .

[7]  D. Rubin,et al.  The central role of the propensity score in observational studies for causal effects , 1983 .

[8]  Russell V. Lenth,et al.  Statistical Analysis With Missing Data (2nd ed.) (Book) , 2004 .

[9]  Brian K. Lee,et al.  Weight Trimming and Propensity Score Weighting , 2011, PloS one.

[10]  Matthias Schonlau,et al.  Selection Bias in Web Surveys and the Use of Propensity Scores , 2006 .

[11]  M. H. Quenouille NOTES ON BIAS IN ESTIMATION , 1956 .

[12]  Roger Tourangeau,et al.  The Science of Web Surveys , 2013 .

[13]  J. N. K. Rao,et al.  Pseudo–Empirical Likelihood Inference for Multiple Frame Surveys , 2010 .

[14]  J. Friedman Stochastic gradient boosting , 2002 .

[15]  Jon A. Krosnick,et al.  Research Synthesis AAPOR Report on Online Panels , 2010 .

[16]  Thomas Lumley,et al.  Analysis of Complex Survey Samples , 2004 .

[17]  Charles DiSogra,et al.  Calibrating Non-Probability Internet Samples with Probability Samples Using Early Adopter Characteristics , 2011 .

[18]  Roger Tourangeau,et al.  Summary Report of the AAPOR Task Force on Non-probability Sampling , 2013 .

[19]  David M. Rothschild,et al.  Forecasting elections with non-representative polls , 2015 .

[20]  Web surveys: methodological problems and research perspectives , 2009 .

[21]  Thomas Lumley,et al.  Complex Surveys: A Guide to Analysis Using R , 2010 .

[22]  Jelke Bethlehem,et al.  The Use of Response Propensities , 2011 .

[23]  Thomas E. Trail,et al.  Hidden Heroes: America's Military Caregivers , 2014 .

[24]  G. Imbens,et al.  Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score , 2000 .

[25]  J. Avorn,et al.  Variable selection for propensity score models. , 2006, American journal of epidemiology.

[26]  Robbert H. Renssen,et al.  Aligning Estimates for Common Variables in Two or More Sample Surveys , 1997 .

[27]  Phillip S. Kott,et al.  Multiple-frame business surveys , 2011 .

[28]  Carl-Erik Särndal,et al.  Generalized Raking Procedures in Survey Sampling , 1993 .

[29]  J. Krosnick,et al.  National Surveys Via Rdd Telephone Interviewing Versus the Internet Comparing Sample Representativeness and Response Quality , 2009 .

[30]  Richard Valliant,et al.  Estimating Propensity Adjustments for Volunteer Web Surveys , 2011 .

[31]  T. Strine,et al.  The PHQ-8 as a measure of current depression in the general population. , 2009, Journal of affective disorders.

[32]  J. Blasius,et al.  Representativeness in Online Surveys through Stratified Samples , 2010 .

[33]  T. Merkouris,et al.  Combining Independent Regression Estimators From Multiple Surveys , 2004 .

[34]  F. Potter,et al.  Methods and Issues in Trimming Extreme Weights in Sample Surveys , 2015 .

[35]  Kosuke Imai,et al.  Survey Sampling , 1998, Nov/Dec 2017.

[36]  Matthias Schonlau,et al.  A Comparison Between Responses From a Propensity-Weighted Web Survey and an Identical RDD Survey , 2004 .

[37]  Phillip S. Kott,et al.  The delete-a-group jackknife: , 2001 .

[38]  J. Hahn On the Role of the Propensity Score in Efficient Semiparametric Estimation of Average Treatment Effects , 1998 .

[39]  Michael Robbins,et al.  A Framework for Synthetic Control Methods With High-Dimensional, Micro-Level Data: Evaluating a Neighborhood-Specific Crime Intervention , 2017 .

[40]  M. H. Quenouille Problems in Plane Sampling , 1949 .

[41]  M. Elliott,et al.  Composite Estimates from Incomplete and Complete Frames for Minimum-Mse Estimation in a Rare Population An Application to Families with Young Children , 2009 .

[42]  D. Rubin,et al.  Statistical Analysis with Missing Data. , 1989 .

[43]  C. Särndal,et al.  Calibration Estimators in Survey Sampling , 1992 .

[44]  Matthias Schonlau,et al.  Are 'Webographic' or Attitudinal Questions Useful for Adjusting Estimates from Web Surveys Using Propensity Scoring? , 2007 .

[45]  J. Friedman Greedy function approximation: A gradient boosting machine. , 2001 .

[46]  D. Yeager,et al.  Comparing the Accuracy of RDD Telephone Surveys and Internet Surveys Conducted with Probability and Non-Probability Samples , 2011 .

[47]  B. Duffy,et al.  Comparing Data from Online and Face-to-face Surveys , 2005 .