Survey method matters: Online/offline questionnaires and face-to-face or telephone interviews differ

Self-report inventories enable efficient assessment of mental attributes in large representative surveys. However, an inventory can be administered in several ways whose equivalence is largely untested. In the present study, we administered thirteen psychological questionnaires assessing positive and negative aspects of mental health. The questionnaires were administered by four different data collection methods: face-to-face interview, telephone interview, online questionnaire, and offline questionnaire. We found that twelve of the questionnaires differed in survey methods. Although, some studies showed that social desirability tends to be highest for telephone survey and lowest for web survey. Furthermore, the effects of social desirability should be the same for the online and offline samples. However, there were no statistically significant differences between the face-to-face and telephone samples for the anxiety scale, the stress scale, and the tradition scale. We also found that for eight scales, the online sample was statistically different from the offline sample in the respondent answers. Moreover, the survey method effects were only moderated by age. Finally, measurement invariance across the four survey methods was tested for each self-report measure. There was full strong measurement invariance established for nine of thirteen scales and partial strong measurement invariance for the remaining four scales across the four survey methods. These findings indicated that measurement invariance was affected by different survey methods. Face-to-face and Telephone samples yielded same results for three scales.Online and Offline samples yielded different results for eight scales.Measurement invariance was affected by different survey methods.

[1]  M. Browne,et al.  Alternative Ways of Assessing Model Fit , 1992 .

[2]  F. Kreuter,et al.  Social Desirability Bias in CATI, IVR, and Web Surveys The Effects of Mode and Question Sensitivity , 2008 .

[3]  Joost C. F. de Winter,et al.  Social desirability is the same in offline, online, and paper surveys: A meta-analysis , 2014, Comput. Hum. Behav..

[4]  Jürgen Margraf,et al.  Personal value orientations as mediated predictors of mental health: A three-culture study of Chinese, Russian, and German university students , 2014, International journal of clinical and health psychology : IJCHP.

[5]  R. Vandenberg,et al.  A Review and Synthesis of the Measurement Invariance Literature: Suggestions, Practices, and Recommendations for Organizational Research , 2000 .

[6]  Jürgen Hoyer,et al.  Die deutsche Version des Life-Orientation-Tests (LOT-R) zum dispositionellen Optimismus und Pessimismus , 2008 .

[7]  E. Brähler,et al.  Fragebogen zur sozialen Unterstützung (F-SozU): Normierung der Kurzform (K-14) , 2009 .

[8]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[9]  R. Larsen,et al.  The Satisfaction with Life Scale , 1985, Journal of personality assessment.

[10]  Ronald E. Anderson,et al.  Computerized School Surveys , 1997 .

[11]  André Beauducel,et al.  On the Performance of Maximum Likelihood Versus Means and Variance Adjusted Weighted Least Squares Estimation in CFA , 2006 .

[12]  J. H. Steiger.  Structural Model Evaluation and Modification: An Interval Estimation Approach. , 1990, Multivariate behavioral research.

[13]  Robert D. Tortora,et al.  Response rate and measurement differences in mixed-mode surveys using mail, telephone, interactive voice response (IVR) and the Internet , 2009 .

[14]  Holger Steinmetz,et al.  Testing measurement invariance using multigroup CFA: differences between educational groups in human values measurement , 2009 .

[15]  S. Schwartz.  Universals in the Content and Structure of Values: Theoretical Advances and Empirical Tests in 20 Countries , 1992 .

[16]  Michael W. Bridges,et al.  Distinguishing optimism from neuroticism (and trait anxiety, self-mastery, and self-esteem): a reevaluation of the Life Orientation Test. , 1994, Journal of personality and social psychology.

[17]  Melanie C. Green,et al.  Telephone versus Face-to-Face Interviewing of National Probability Samples with Long Questionnaires: Comparisons of Respondent Satisficing and Social Desirability Response Bias , 2003 .

[18]  H. Young,et al.  Development and psychometric evaluation of the Resilience Scale. , 1993, Journal of nursing measurement.

[19]  Ronald Fischer,et al.  Testing measurement invariance across groups: applications in cross-cultural research. , 2010 .

[20]  Mark D. Griffiths,et al.  ONLINE FORUMS AND SOLICITED BLOGS: INNOVATIVE METHODOLOGIES FOR ONLINE GAMING DATA COLLECTION , 2016 .

[21]  S. Kiesler,et al.  Response Effects in the Electronic Survey , 1986 .

[22]  Noriko Yoshimura,et al.  Descriptive Epidemiology of Somatising Tendency: Findings from the CUPID Study , 2016, PloS one.

[23]  A. Antonovsky.  Unraveling the mystery of health: how people manage stress and stay well , 1987 .

[24]  Jörg Schumacher,et al.  Die Resilienzskala - Ein Fragebogen zur Erfassung der psychischen Widerstandsfähigkeit als Personmerkmal. , 2005 .

[25]  Herschel Knapp,et al.  Using pencil and paper, Internet and touch-tone phones for self-administered surveys: does methodology matter? , 2003, Comput. Hum. Behav..

[26]  B. Byrne,et al.  Testing for the equivalence of factor covariance and mean structures: The issue of partial measurement invariance. , 1989 .

[27]  Chao Wen,et al.  An assessment of equivalence between paper and social media surveys: The role of social desirability and satisficing , 2014, Comput. Hum. Behav..

[28]  S. Lyubomirsky,et al.  A Measure of Subjective Happiness: Preliminary Reliability and Construct Validation , 1999 .

[29]  Paul Rosenfeld,et al.  Impression management, social desirability, and computer administration of attitude questionnaires: Does the computer make a difference? , 1992 .

[30]  S. Booth-Kewley,et al.  Computer-Administered Surveys in Organizational Settings , 1993 .

[31]  Kristopher J Preacher,et al.  Sample Size in Factor Analysis: The Role of Model Error , 2001, Multivariate behavioral research.

[32]  Jürgen Margraf,et al.  Social Rhythm and Mental Health: A Cross-Cultural Comparison , 2016, PloS one.

[33]  D. Flora,et al.  An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. , 2004, Psychological methods.

[34]  Y. Torres,et al.  Anxious and non-anxious major depressive disorder in the World Health Organization World Mental Health Surveys , 2015, Epidemiology and Psychiatric Sciences.

[35]  P. Lachenbruch.  Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[36]  Eldad Davidov,et al.  Testing for measurement equivalence of human values across online and paper-and-pencil surveys , 2011 .

[37]  Jörg Schumacher,et al.  Deutsche Normierung der Sense of Coherence Scale von Antonovsky , 2000 .

[38]  Eni S. Becker,et al.  Psychometric properties of the Positive Mental Health Scale (PMH-scale) , 2016, BMC Psychology.

[39]  Aike Hessel,et al.  Fragebogen zur Sozialen Unterstützung (F-SozU) , 1999 .

[40]  P. Bentler,et al.  Comparative fit indexes in structural models. , 1990, Psychological bulletin.

[41]  Dieter Hoffmann,et al.  Online, face-to-face and telephone surveys—Comparing different sampling methods in wine consumer research , 2013 .

[42]  J. Krosnick,et al.  National Surveys Via Rdd Telephone Interviewing Versus the Internet Comparing Sample Representativeness and Response Quality , 2009 .