Methodological Considerations in Using Complex Survey Data

Complex survey data are collected by means other than simple random samples. This creates two analytical issues: nonindependence and unequal selection probability. Failing to address these issues results in underestimated standard errors and biased parameter estimates. Using data from the nationally representative Head Start Family and Child Experiences Survey (FACES; 1997 and 2000 cohorts), three diverse multilevel models are presented that illustrate differences in results depending on addressing or ignoring the complex sampling issues. Limitations of using complex survey data are reported, along with recommendations for reporting complex sample results.

[1]  H. Goldstein,et al.  Weighting for unequal selection probabilities in multilevel models , 1998 .

[2]  Scott L. Thomas,et al.  Weighting and adjusting for design effects in secondary data analyses , 2005 .

[3]  M. Appelbaum,et al.  Some issues of conducting secondary analyses , 1991 .

[4]  K. Wolter.  Introduction to Variance Estimation , 1985 .

[5]  A. Satorra,et al.  Complex Sample Data in Structural Equation Modeling , 1995 .

[6]  Aaron J. Ferguson,et al.  On the Utilization of Sample Weights in Latent Variable Models. , 1999 .

[7]  Stephen W. Raudenbush,et al.  Effects of Kindergarten Retention Policy on Children’s Cognitive Growth in Reading and Mathematics , 2005 .

[8]  Debbie L. Hahs-Vaughn,et al.  National profiles of school readiness skills for Head Start children: An investigation of stability and change , 2012 .

[9]  G. Pike.  Using Weighting Adjustments to Compensate for Survey Nonresponse , 2008 .

[10]  Debbie L. Hahs-Vaughn.  Weighting Omissions and Best Practices When Using Large-Scale Data in Educational Research , 2006 .

[11]  P. J. McCarthy,et al.  Replication: an approach to the analysis of data from complex surveys. , 1966, Vital and health statistics. Series 2, Data evaluation and methods research.

[12]  Debbie L. Hahs-Vaughn,et al.  National Profiles of classroom quality and family involvement: A multilevel examination of proximal influences on Head Start children's school readiness , 2012 .

[13]  Eun Sul Lee,et al.  Analyzing Complex Survey Data , 1989 .

[14]  Graham Kalton,et al.  Introduction to Survey Sampling , 1983 .

[15]  L. Kish,et al.  Inference from Complex Samples , 1974 .

[16]  Graham Kalton,et al.  Models in the Practice of Survey Sampling , 1983 .

[17]  Ita G. G. Kreft,et al.  Multilevel Analysis Methods , 1994 .

[18]  P. Mahalanobis.  On large-scale sample surveys , 1944, Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences.

[19]  Jeffrey D. Kromrey,et al.  Alternatives for Analysis of Complex Sample Surveys: A Comparison of SAS ® , SUDAAN ® , and AM ® Software , 2007 .

[20]  N. Schenker,et al.  On Judging the Significance of Differences by Examining the Overlap Between Confidence Intervals , 2001 .

[21]  Debbie L. Hahs-Vaughn.  Analysis of data from complex samples , 2006 .

[22]  Kenneth G. Manton,et al.  “Equivalent Sample Size” and “Equivalent Degrees of Freedom” Refinements for Inference Using Survey Weights under Superpopulation Models , 1992 .

[23]  S. Hofferth.  Secondary Data Analysis in Family Research , 2005 .

[24]  Thomas Lumley,et al.  Analysis of Complex Survey Samples , 2004 .

[25]  Anthony S. Bryk,et al.  Hierarchical Linear Models: Applications and Data Analysis Methods , 1992 .

[26]  Debbie L. Hahs-Vaughn,et al.  A Primer for Using and Understanding Weights With National Datasets , 2005 .

[27]  Laura M. Stapleton,et al.  The Incorporation of Sample Weights Into Multilevel Structural Equation Models , 2002 .

[28]  J. R. Landis,et al.  A statistical methodology for analyzing data from a complex survey: the first National Health and Nutrition Examination Survey. , 1982, Vital and health statistics. Series 2, Data evaluation and methods research.

[29]  Edward S. Cavin,et al.  An Application of Balanced Repeated Replication (Brr) Variance Estimation To Program Evaluation , 1990 .

[30]  Michael J. Keane,et al.  A Descriptive Study of Head Start Families: FACES Technical Report I. , 2002 .

[31]  Paul P. Biemer,et al.  Weighting survey data , 2008 .

[32]  K F Rust,et al.  Variance estimation for complex surveys using replication techniques , 1996, Statistical methods in medical research.

[33]  Sameena Salvucci,et al.  National Education Longitudinal Study of 1988 (NELS:88) Research Framework and Issues. Working Paper Series. , 1996 .

[34]  Richard G. Lomax,et al.  Utilization of Sample Weights in Single-Level Structural Equation Modeling , 2006 .

[35]  W. DuMouchel,et al.  Using Sample Survey Weights in Multiple Regression Analyses of Stratified Samples , 1983 .

[36]  J. N. K. Rao,et al.  Bootstrap and other methods to measure errors in survey estimates , 1988 .

[37]  Stanley Lemeshow,et al.  Estimating the variances of ratio estimates in complex sample surveys with two primary sampling units per stratum—a comparison of balanced replication and jackknife techniques , 1979 .

[38]  Daniel J Bauer,et al.  Local solutions in the estimation of growth mixture models. , 2006, Psychological methods.

[39]  Chris J. Skinner,et al.  Analysis of complex surveys , 1991 .

[40]  Laura M. Stapleton,et al.  An Assessment of Practical Solutions for Structural Equation Modeling with Complex Sample Data , 2006 .

[41]  D. Dillman,et al.  International handbook of survey methodology. , 2008 .

[42]  Annemarie H. Hindman,et al.  Ecological contexts and early learning: Contributions of child, family, and classroom factors during Head Start, to literacy and mathematics growth through first grade , 2010 .

[43]  Edward L. Korn,et al.  Examples of Differing Weighted and Unweighted Estimates from a Sample Survey , 1995 .

[44]  Scott L. Thomas,et al.  Analysis of Large-Scale Secondary Data in Higher Education Research: Potential Perils Associated with Complex Sampling Designs , 2001 .