Insufficient Effort Survey Responding: An Under‐Appreciated Problem in Work and Organisational Health Psychology Research

Insufficient effort responding (IER) is problematic in that it can add a systematic source of variance for variables with average responses that depart from the scale midpoints. We present a rationale for why IER is of particular importance to Work and Organisational Health Psychology (WOHP) researchers. We also demonstrate its biasing effects using several variables of interest to WOHP researchers (perceived work ability, negative affectivity, perceived disability, work–safety tension, accident/injury frequencies, and experienced and instigated incivility) in two datasets. As expected, IER was significantly correlated with the focal study variables. We also found some evidence that hypothesised bivariate correlations between these variables were inflated when IER respondents were included. Corroborating IER's potential confounding role, we further found significant declines in the magnitude of the hypothesised bivariate correlations after partialling out IER. In addition, we found evidence for biasing (under-estimation) effects for predictors not contaminated by IER in multiple regression models where some predictors and the outcome were both contaminated by IER. We call for WOHP researchers to routinely discourage IER from occurring in their surveys, screen for IER prior to analyzing survey data, and establish a standard practice for handling IER cases.

[1]  Pauline Schilpzand,et al.  Workplace incivility: A review of the literature and agenda for future research , 2016 .

[2]  Jason L. Huang,et al.  Detecting Insufficient Effort Responding with an Infrequency Scale: Evaluating Validity and Participant Reactions , 2014, Journal of Business and Psychology.

[3]  Jason L. Huang,et al.  Difference in Response Effort Across Sample Types: Perception or Reality? , 2015, Industrial and Organizational Psychology.

[4]  Jason L. Huang,et al.  Insufficient effort responding: examining an insidious confound in survey data. , 2015, The Journal of applied psychology.

[5]  Reanna Poncheri Harman,et al.  Exploring qualitative training reactions: individual and contextual influences on trainee commenting. , 2015, The Journal of applied psychology.

[6]  Larissa K. Barber,et al.  Turning molehills into mountains: Sleepiness increases workplace interpretive bias , 2015 .

[7]  Richard N. Landers,et al.  An Inconvenient Truth: Arbitrary Distinctions Between Organizational, Mechanical Turk, and Other Convenience Samples , 2015, Industrial and Organizational Psychology.

[8]  Alyssa K. McGonagle,et al.  Individual and work factors related to perceived work ability and labor force outcomes. , 2015, The Journal of applied psychology.

[9]  M. K. Ward Careless Responding on Internet-Based Surveys , 2015 .

[10]  Jason L. Huang Does cleanliness influence moral judgments? Response effort moderates the effect of cleanliness priming on moral judgments , 2014, Front. Psychol..

[11]  V. Lim,et al.  Applied Psychology: An International Review Special Issue , 2014 .

[12]  Benjamin M. Walsh,et al.  Civility norms, safety climate, and safety outcomes: a preliminary investigation. , 2014, Journal of occupational health psychology.

[13]  Robert LaRose,et al.  Completion rates and non-response error in online surveys: Comparing sweepstakes and pre-paid cash incentives in studies of online behavior , 2014, Comput. Hum. Behav..

[14]  Russell A. Matthews,et al.  An eye for an eye? Exploring the relationship between workplace incivility experiences and perpetration. , 2014, Journal of occupational health psychology.

[15]  D. Rosenthal,et al.  Quality-space theory in olfaction , 2014, Front. Psychol..

[16]  D. Simons The Value of Direct Replication , 2014, Perspectives on psychological science : a journal of the Association for Psychological Science.

[17]  H. Zacher,et al.  The moderating effects of job control and selection, optimization, and compensation strategies on the age-work ability relationship , 2013 .

[18]  Karsten Mueller,et al.  Nonresponse in Employee Attitude Surveys: A Group-Level Analysis , 2013 .

[19]  Benjamin M. Walsh,et al.  Assessing Workgroup Norms for Civility: The Development of the Civility Norms Questionnaire-Brief , 2012 .

[20]  C. Piotrowski Occupational Health Psychology: Neglected Areas of Research , 2012 .

[21]  A. Meade,et al.  Identifying careless responses in survey data. , 2012, Psychological methods.

[22]  S. B. Button,et al.  Rethinking carelessness on job analysis surveys: Not all questions are created equal. , 2012 .

[23]  T. Allen,et al.  Relationship between Work Interference with Family and Parent-Child Interactive Behavior: Can Guilt Help?. , 2012 .

[24]  P. Jose,et al.  Identity, Perceived Religious Discrimination, and Psychological Well‐Being in Muslim Immigrant Women , 2012 .

[25]  Elizabeth M. Poposki,et al.  Detecting and Deterring Insufficient Effort Responding to Surveys , 2012 .

[26]  Karsten Mueller,et al.  On the relationship between job satisfaction and non-response in employee attitude surveys: A longitudinal field study , 2011 .

[27]  Daniel B. Goldman,et al.  Rumination About an Interpersonal Offense Scale , 2011 .

[28]  V. Magley,et al.  Sensitivity to interpersonal treatment in the workplace: Scale development and initial validation: Sensitivity to interpersonal treatment , 2011 .

[29]  C. Nygård,et al.  Work ability in midlife as a predictor of mortality and disability in later life: a 28-year prospective follow-up study , 2011, Canadian Medical Association Journal.

[30]  Lisa M. Kath,et al.  Work-safety tension, perceived risk, and worker injuries: a meso-mediational model. , 2010, Journal of safety research.

[31]  Stephanie L. Morrow,et al.  Relationships between psychological safety climate facets and safety behavior in the rail industry: a dominance analysis. , 2010, Accident; analysis and prevention.

[32]  Dov Zohar,et al.  Thirty years of safety climate research: reflections and future directions. , 2010, Accident; analysis and prevention.

[33]  Evangelia Demerouti,et al.  Burnout and work engagement: a thorough investigation of the independency of both constructs. , 2010, Journal of occupational health psychology.

[34]  R. McGrath,et al.  Evidence for response bias as a source of error variance in applied assessment. , 2010, Psychological bulletin.

[35]  L. Rehm International Association of Applied Psychology , 2010 .

[36]  Jill C. Bradley,et al.  Workplace safety: a meta-analysis of the roles of person and situation factors. , 2009, The Journal of applied psychology.

[37]  L. Cortina,et al.  Patterns and profiles of response to incivility in the workplace. , 2009, Journal of occupational health psychology.

[38]  T. Taris,et al.  How may nonresponse affect findings in organizational surveys? The tendency-to-the-positive effect , 2007 .

[39]  S. Marklund,et al.  The relationship between perceived employability and subsequent health , 2007 .

[40]  Angela L. Duckworth,et al.  Grit: perseverance and passion for long-term goals. , 2007, Journal of personality and social psychology.

[41]  S. Rogelberg,et al.  Introduction Understanding and Dealing With Organizational Survey Nonresponse , 2007 .

[42]  D. Hofmann,et al.  An investigation of the relationship between safety climate and medication errors as well as other nurse and patient outcomes. , 2006 .

[43]  John A. Johnson,et al.  The international personality item pool and the future of public-domain personality measures ☆ , 2006 .

[44]  Gary Blau,et al.  Testing a measure of instigated workplace incivility , 2005 .

[45]  Michael E. Brown,et al.  Ethical leadership: A social learning perspective for construct development and testing , 2005 .

[46]  Gilad Chen,et al.  Examination of the Relationships among General and Work-Specific Self-Evaluations, Work-Related Control Beliefs, and Job Attitudes , 2004 .

[47]  Gilad Chen,et al.  General self-efficacy and self-esteem: toward theoretical and empirical distinction between correlated self-evaluations , 2004 .

[48]  Scott B. MacKenzie,et al.  Common method biases in behavioral research: a critical review of the literature and recommended remedies. , 2003, The Journal of applied psychology.

[49]  E. Kelloway,et al.  Development and test of a model linking safety-specific transformational leadership and occupational safety. , 2002, The Journal of applied psychology.

[50]  Kibeom Lee,et al.  Organizational citizenship behavior and workplace deviance: the role of affect and cognitions. , 2002, The Journal of applied psychology.

[51]  G. Ironson,et al.  A General Measure of Work Stress: The Stress in General Scale , 2001 .

[52]  D. Zohar A group-level model of safety climate: testing the effect of group climate on microaccidents in manufacturing jobs. , 2000, The Journal of applied psychology.

[53]  Paul E. Spector,et al.  Why negative affectivity should not be controlled in job stress research: don't throw out the baby with the bath water , 2000, Journal of Organizational Behavior.

[54]  Kelly L. Zellars,et al.  An examination of attributions and emotions in the transactional approach to the organizational stress process , 1999 .

[55]  Christine M. Pearson,et al.  Tit for Tat? The Spiraling Effect of Incivility in the Workplace , 1999 .

[56]  Bob E. Hayes,et al.  Measuring Perceptions of Workplace Safety: Development and Validation of the Work Safety Scale , 1998 .

[57]  Steven G. Rogelberg,et al.  Nonresponse to Mailed Surveys , 1998 .

[58]  D. Vandewalle Development and Validation of a Work Domain Goal Orientation Instrument , 1997 .

[59]  Paul E. Spector Using self‐report questionnaires in OB research: A comment on the use of a controversial method , 1994 .

[60]  L. J. Williams,et al.  An Alternative Approach to Method Effects by Using Latent-Variable Models: Applications in Organizational Behavior Research , 1994 .

[61]  François Béland,et al.  A safety climate measure for construction sites , 1991 .

[62]  Bor-Shiuan Cheng,et al.  Cultural Relativity in Action: A Comparison of Self-Ratings Made by Chinese and U.S. Workers , 1991 .

[63]  P. Hewitt,et al.  Perfectionism in the self and social contexts: conceptualization, assessment, and association with psychopathology. , 1991, Journal of personality and social psychology.

[64]  Newell K. Eaton,et al.  Criterion-related validities of personality constructs and the effect of response distortion on those validities , 1990 .

[65]  S. Ashford,et al.  Content, Cause, and Consequences of Job Insecurity: A Theory-Based Measure and Substantive Test , 1989 .

[66]  S. Hobfoll Conservation of resources. A new attempt at conceptualizing stress. , 1989, The American psychologist.

[67]  D. Watson,et al.  Development and validation of brief measures of positive and negative affect: the PANAS scales. , 1988, Journal of personality and social psychology.

[68]  R. Eisenberger,et al.  Perceived organizational support. , 1986 .

[69]  T. Sterling,et al.  Extent, persistence, and constancy of the healthy worker or healthy person effect by all and selected causes of death. , 1986, Journal of occupational medicine. : official publication of the Industrial Medical Association.

[70]  L. Derogatis,et al.  The Brief Symptom Inventory: an introductory report , 1983, Psychological Medicine.

[71]  D. Zohar Safety climate in industrial organizations: theoretical and applied implications. , 1980, The Journal of applied psychology.

[72]  P. Converse,et al.  The Quality of American Life: Perceptions, Evaluations, and Satisfactions , 1976 .

[73]  Kathleen C. Gerbasi,et al.  Short, homogeneous versions of the Marlow‐Crowne Social Desirability Scale , 1972 .

[74]  C. Spearman General intelligence Objectively Determined and Measured , 1904 .

[75]  Jason L. Huang,et al.  Insufficient Effort Responding to Surveys as a Threat to Validity: The Perceptions and Practices of SIOP Members , 2013 .

[76]  Paul E. Spector,et al.  Quantitative self-report methods in occupational health psychology research , 2013 .

[77]  Scott B. MacKenzie,et al.  Working memory: theories, models, and controversies. , 2012, Annual review of psychology.

[78]  A. Baddeley Working memory: theories, models, and controversies. , 2012, Annual review of psychology.

[79]  Seth A. Kaplan,et al.  Workplace safety and accidents: An industrial and organizational psychology perspective. , 2011 .

[80]  T. B. Üstün,et al.  Measuring health and disability : manual for WHO Disability Assessment Schedule : WHODAS 2.0 , 2010 .

[81]  J. Ilmarinen,et al.  Diversity of work ability , 2008 .

[82]  V. Anne Smith,et al.  Testing measures of animal social association by computer simulation , 2007 .

[83]  Philip Bobko,et al.  Content , Causes , and Consequences of Job Insecurity : A Theory-Based Measure and Substantive Test , 2007 .

[84]  J. Attia,et al.  The healthy worker survivor effect in a study of neck muscle performance measures in call-centre operators. , 2006, Work.

[85]  E. ElizabethMannix,et al.  Diversity at Work , 2006 .

[86]  L. Cortina,et al.  Incivility in the workplace: incidence and impact. , 2001, Journal of occupational health psychology.

[87]  J. Bermúdez,et al.  Personality Psychology in Europe , 1997 .

[88]  Cortlandt Cammann,et al.  Assessing the attitudes and perceptions of organizational members , 1983 .