The Use of a Nonprobability Internet Panel to Monitor Sexual and Reproductive Health in the General Population

Background: Reliability of nonprobability online volunteer panels for epidemiological purposes has rarely been studied. Objectives: To assess the quality of a questionnaire on sexual and reproductive health (SRH) administered in a nonprobability Web panel and in a random telephone survey (n = 8,992; n = 8,437, age 16–49 years). Especially, we were interested in the possible difference in the association of sociodemographic variables and some outcome variables in the two surveys that are in the reliability of analytical epidemiological studies conducted in such panels. Methods: Interventions to increase response rate were used in both surveys (four e-mail reminders, high number of call attempts and callbacks to refusals). Both were calibrated on the census population. Sociodemographic composition, effects of reminders, and prevalence were compared to their telephone counterpart. In addition, the associations of sociodemographic and sexual behaviors were compared in the two samples in multivariate logistic regressions. Results: The online survey had a lower response rate (20.0 percent vs. 44.8 percent) and a more distorted sociodemographic structure although the reminders improved the representativeness as did the analogous interventions on the telephone survey. Prevalences of SRH variables were similar for the common behaviors but higher online for the stigmatized behaviors, depending on gender. Overall, 29 percent of the 63 interactions studied were significant for males and 11 percent for women, although opposite effects of sociodemographic variables were rare (5 percent of the 171 tested for each gender). Conclusion: Nonprobability online panels are to be used with caution to monitor SRH and conduct analytical epidemiological studies, especially among men.

[1]  P. Chisnall.  Mail and Internet Surveys: The Tailored Design Method , 2007, Journal of Advertising Research.

[2]  Carina Paine,et al.  Responding to sensitive questions in surveys : A comparison of results from Online panels , face to face , and self-completion interviews , 2008 .

[3]  A. Schmidt,et al.  Prevalence of HIV among MSM in Europe: comparison of self-reported diagnoses from a large scale internet survey and existing national estimates , 2012, BMC Public Health.

[4]  K. Wellings,et al.  Sexual behaviour in context: a global perspective , 2006, The Lancet.

[5]  P. Peretti-Watel,et al.  Les usages de drogues illicites déclarés par les adolescents selon le mode de collecte , 2001 .

[6]  Til Stürmer,et al.  A review of the application of propensity score methods yielded increasing use, advantages in specific settings, but not substantially different estimates compared with conventional multivariable methods. , 2006, Journal of clinical epidemiology.

[7]  D. Dillman,et al.  International handbook of survey methodology. , 2008 .

[8]  V. Gombault.  L'internet de plus en plus prisé, l'internaute de plus en plus mobile , 2013 .

[9]  Jelke Bethlehem,et al.  Indicators for the representativeness of survey response , 2009 .

[10]  Accounting for the Effects of Data Collection Method: Application to the International Tobacco Control Netherlands Survey , 2014 .

[11]  Roger Tourangeau,et al.  Summary Report of the AAPOR Task Force on Non-probability Sampling , 2013 .

[12]  Victor Aguirre-Torres.  THE EFFECT AND ADJUSTMENT OF COMPLEX SURVEYS ON CHI-SQUARED GOODNESS OF FIT TESTS , 2002 .

[13]  R. Loo,et al.  Confirmatory Factor Analyses of the Full and Short Versions of the Marlowe-Crowne Social Desirability Scale , 2000, The Journal of social psychology.

[14]  Mario Callegaro,et al.  Computing Response Metrics for Online Panels , 2008 .

[15]  Jan-Eric Litton,et al.  New times, new needs; e-epidemiology , 2007, European Journal of Epidemiology.

[16]  M. Couper.  A REVIEW OF ISSUES AND APPROACHES , 2000 .

[17]  Gergana Y. Nenkov,et al.  A short form of the Maximization Scale: Factor structure, reliability and validity studies , 2008, Judgment and Decision Making.

[18]  Kaye Wellings,et al.  Nonprobability Web Surveys to Measure Sexual Behaviors and Attitudes in the General Population: A Comparison With a Probability Sample Interview Survey , 2014, Journal of medical Internet research.

[19]  Vie personnelle et enquête par téléphone : l'exemple de l'enquête ACSF , 1993 .

[20]  C. Rossier.  Estimating induced abortion rates: a review. , 2003, Studies in family planning.

[21]  F. Kreuter,et al.  Social Desirability Bias in CATI, IVR, and Web Surveys The Effects of Mode and Question Sensitivity , 2008 .

[22]  Richard Valliant,et al.  Internet Surveys: Can Statistical Adjustments Eliminate Coverage Bias? , 2008 .

[23]  Maria Varedian.  Comparing propensity score weighting with other Weighting Methods : A Case Study , 2003 .

[24]  Richard Valliant,et al.  Estimating Propensity Adjustments for Volunteer Web Surveys , 2011 .

[25]  A. Mackley,et al.  A Novel Approach to Improving Current Research Awareness: A Web-Based Nursing Research Journal Watch , 2010, Journal for nurses in staff development : JNSD : official journal of the National Nursing Staff Development Organization.

[26]  G. Loosveldt,et al.  An evaluation of the weighting procedures for an online access panel survey , 2008 .

[27]  David Haziza,et al.  Imputation and Inference in the Presence of Missing Data , 2009 .

[28]  Sunghee Lee.  Propensity score adjustment as a weighting scheme for volunteer panel web surveys , 2006 .

[29]  D. Rubin,et al.  Reducing Bias in Observational Studies Using Subclassification on the Propensity Score , 1984 .

[30]  F. Beck,et al.  Does Computer Survey Technology Improve Reports on Alcohol and Illicit Drug Use in the General Population? A Comparison Between Two Surveys with Different Data Collection Modes In France , 2014, PloS one.

[31]  Edith D. de Leeuw,et al.  Choosing the Method of Data Collection , 2008 .

[32]  Matthias Schonlau,et al.  Are 'Webographic' or Attitudinal Questions Useful for Adjusting Estimates from Web Surveys Using Propensity Scoring? , 2007 .

[33]  Melanie C. Green,et al.  Telephone versus Face-to-Face Interviewing of National Probability Samples with Long Questionnaires: Comparisons of Respondent Satisficing and Social Desirability Response Bias , 2003 .

[34]  E. Weiderpass,et al.  Feasibility of Using Web-based Questionnaires in Large Population-based Epidemiological Studies , 2005, European Journal of Epidemiology.

[35]  A. Scott.  Rao-Scott corrections and their impact , 2007 .

[36]  Sunghee Lee,et al.  Estimation for Volunteer Panel Web Surveys Using Propensity Score Adjustment and Calibration Adjustment , 2009 .

[37]  D. Rubin,et al.  The central role of the propensity score in observational studies for causal effects , 1983 .

[38]  Christian Boudreau,et al.  Mesurer les effets de la méthode de collecte des données , 2013 .

[39]  J. Krosnick.  Response strategies for coping with the cognitive demands of attitude measures in surveys , 1991 .

[40]  P. Austin.  An Introduction to Propensity Score Methods for Reducing the Effects of Confounding in Observational Studies , 2011, Multivariate behavioral research.

[41]  Geert Loosveldt,et al.  Face-to-Face versus Web Surveying in a High-Internet-Coverage Population Differences in Response Quality , 2008 .

[42]  Michael W. Link,et al.  Alternative Modes for Health Surveillance Surveys: An Experiment with Web, Mail, and Telephone , 2005, Epidemiology.

[43]  S. Legleye,et al.  Improving Survey Participation Cost Effectiveness of Callbacks to Refusals and Increased Call Attempts in a National Telephone Survey in France , 2013 .

[44]  Matthias Schonlau,et al.  A Comparison Between Responses From a Propensity-Weighted Web Survey and an Identical RDD Survey , 2004 .

[45]  Matthias Schonlau,et al.  Selection Bias in Web Surveys and the Use of Propensity Scores , 2006 .

[46]  Sunghee Lee,et al.  Simple Approaches to Estimating the Variance of the Propensity Score Weighted Estimator Applied on Volunteer Panel Web Survey Data – a Comparative Study , 2005 .

[47]  C. Moreau,et al.  La contraception en France : nouveau contexte, nouvelles pratiques ? , 2012, Population & Sociétés.