Sampling Bias and Other Methodological Threats to the Validity of Health Survey Research

Data from a longitudinal occupational health survey of professional fire fighters were used to explore the potential impact of two types of methodological bias: sample selection and reactivity. No significant differences on demographic variables were observed between the group who first responded after a within-study change in survey administration format (Delayed) and respondents who had completed surveys since the study's inception (Initial). However, statistically significant differences in the study's 26 outcome measures provided some evidence that between-group differences did exist and that an “administration format” type of response bias was also potentially present. The effect sizes associated with the 37 observed significant differences ranged from small to medium. These results provide a context for a reexamination of standard techniques for the identification and interpretation of survey research biases. Methods are suggested to strengthen tests for selection bias and to minimize the impact of response biases.

[1]  J. Luxenberg,et al.  Misclassification and Selection Bias When Identifying Alzheimer's Disease Solely from Medicare Claims Records , 1999, Journal of the American Geriatrics Society.

[2]  D. W. Zimmerman,et al.  Are Simple Gain Scores Obsolete? , 1996 .

[3]  Jack Block,et al.  The challenge of response sets: Unconfounding meaning, acquiescence, and social desirability in the MMPI. , 1965 .

[4]  P. Lachenbruch.  Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[5]  D. Morgan,et al.  Use of Public Records to Compare Respondents and Nonrespondents in a Study of Recent Widows , 1996 .

[6]  T. Raykov.  Are Simple Change Scores Obsolete? An Approach to Studying Correlates and Predictors of Change , 1999 .

[7]  R Gist,et al.  Occupational stress in contemporary fire service. , 1995, Occupational medicine.

[8]  R. Rosenthal.  Parametric measures of effect size. , 1994 .

[9]  L. Carstensen,et al.  Social desirability and the measurement of psychological well-being in elderly persons. , 1983, Journal of gerontology.

[10]  D. Watson,et al.  The vicissitudes of mood measurement: effects of varying descriptors, time frames, and response formats on measures of positive and negative affect. , 1988, Journal of personality and social psychology.

[11]  M. Wiederman.  Volunteer bias in sexuality research using college student participants , 1999 .

[12]  D. Relles,et al.  Tools for intuition about sample selection bias and its correction , 1997 .

[13]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[14]  Kenneth M. Keating.  Self-selection: Are we beating a dead horse? , 1989 .

[15]  R. Berk.  An introduction to sample selection bias in sociological data. , 1983 .

[16]  J. Walsh.  Comment on social desirability. , 1990 .

[17]  William G. Cochran,et al.  Experimental Designs, 2nd Edition , 1950 .

[18]  Allen L. Edwards,et al.  The Relationship Between the Judged Desirability of a Trait and the Probability That the Trait Will Be Endorsed , 1953 .

[19]  A. L. Edwards.  The measurement of personality traits by scales and inventories , 1970 .

[20]  A. L. Edwards.  The social desirability variable in personality assessment and research , 1958 .

[21]  David Watkins,et al.  Culture, Gender, and Response Bias , 1995 .

[22]  D. Watson,et al.  Development and validation of brief measures of positive and negative affect: the PANAS scales. , 1988, Journal of personality and social psychology.

[23]  R. Beaton,et al.  Self-reported symptoms of stress with temporomandibular disorders: comparisons to healthy men and women. , 1991, The Journal of prosthetic dentistry.

[24]  L. Hedges,et al.  The Handbook of Research Synthesis , 1995 .

[25]  G. J. Mellenbergh.  A Note on Simple Gain Score Precision , 1999 .

[26]  William G. Cochran,et al.  Experimental designs, 2nd ed. , 1957 .

[27]  D. Campbell,et al.  EXPERIMENTAL AND QUASI-EXPERIMENT Al DESIGNS FOR RESEARCH , 2012 .

[28]  G. Glass.  Primary, Secondary, and Meta-Analysis of Research1 , 1976 .

[29]  H. Turner.  Participation bias in AIDS‐related telephone surveys: Results from the national AIDS behavioral survey (NABS) non‐response study , 1999 .

[30]  M. Berk,et al.  Can Survey Data Be Used to Estimate Physician Practice Costs? , 1996, Evaluation & the health professions.

[31]  H. Kogan,et al.  Self‐management: a nursing mode of therapeutic influence , 1984, ANS. Advances in nursing science.

[32]  J. Catania.  A framework for conceptualizing reporting bias and its antecedents in interviews assessing human sexuality , 1999 .

[33]  More remarks on social desirability. , 1990 .

[34]  Christopher Winship,et al.  Models for Sample Selection Bias , 1992 .

[35]  Steven L. Sauter,et al.  Organizational Risk Factors for Job Stress , 1995 .

[36]  Paul E. Spector,et al.  The role of negative affectivity in employee reactions to job characteristics: Bias effect or substantive effect? , 1999 .

[37]  Paul T. Costa,et al.  Social desirability scales: More substance than style. , 1983 .

[38]  A. Valdez,et al.  Reducing Selection Bias in the Use of Focus Groups to Investigate Hidden Populations: The Case of Mexican-American Gang Members from South Texas , 1998, Conducting Drug Abuse Research with Minority Populations: Advances and Issues.

[39]  U. John,et al.  Estimating prevalence of alcohol abuse and dependence in one general hospital: an approach to reduce sample selection bias. , 1999, Alcohol and alcoholism.

[40]  W. Corneil,et al.  Exposure to Duty-Related Incident Stressors in Urban Firefighters and Paramedics , 1998, Journal of traumatic stress.

[41]  Mark G. Wilson,et al.  A Comprehensive Review of the Effects of Worksite Health Promotion on Health-Related Outcomes , 1996, American journal of health promotion : AJHP.

[42]  L. Collins.  Is Reliability Obsolete? A Commentary on "Are Simple Gain Scores Obsolete?" , 1996 .

[43]  R. Ellsworth.  Does Follow-Up Loss Reflect Poor Outcome? , 1979, Evaluation & the health professions.

[44]  James S. Phillips,et al.  Subject Recruitment Bias: The Paid Volunteer Subject , 1978 .

[45]  A. L. Edwards.  Construct validity and social desirability. , 1990 .

[46]  Kent B. Monroe,et al.  Effect-Size Estimates: Issues and Problems in Interpretation , 1996 .

[47]  Sanford L. Braver,et al.  Assessing and Compensating for Self-Selection Bias (Non-representativeness) of the Family Research Sample. , 1992 .

[48]  L. Humphreys.  Linear Dependence of Gain Scores on Their Components Imposes Constraints on Use and Interpretation: Comment on "Are Simple Gain Scores Obsolete?" , 1996 .

[49]  E. Hudes,et al.  Estimating and correcting for response bias in self-reported HIV risk behavior. , 1999 .