Questionnaire layout and national culture in online psychometrics

Given the proliferation of online psychometric questionnaires on the Internet and other platforms, the design of online psychometrics becomes increasingly important to ensure good measurement properties. The current study reports four experiments testing how questionnaire layout and national culture affect people׳s responses to online questionnaires. Flow- and the disorientation scales for web navigation had good psychometric quality overall and across experimental manipulations of questionnaire layout, field dependence, national culture, response correction and question grouping. However, single-item layout (presenting one question at a time) had the advantage of faster completion than whole-form layout (presenting each page filled with items). Support was also found for the idea that whole form divides and thereby requires more attention to respond. Future research should be directed at online psychometrics on small and large displays.

[1]  Fred D. Davis,et al.  A critical assessment of potential measurement biases in the technology acceptance model: three experiments , 1996, Int. J. Hum. Comput. Stud..

[2]  J. Roldán,et al.  Applying Maximum Likelihood and PLS on Different Sample Sizes: Studies on SERVQUAL Model and Employee Behavior Model , 2010 .

[3]  G Houghton,et al.  Word-based grouping affects the prime-task effect on semantic priming. , 2000, Journal of experimental psychology. Human perception and performance.

[4]  M. J. Emerson,et al.  Field dependence–independence from a working memory perspective: A dual-task investigation of the Hidden Figures Test , 2001, Memory.

[5]  Roger Tourangeau,et al.  The impact of progress indicators on task completion , 2010, Interact. Comput..

[6]  Xingti. Liu,et al.  Assessing Measurement Invariance of the Teachers’ Perceptions of Grading Practices Scale across Cultures , 2008 .

[7]  G. Wulf,et al.  Effortless motor learning?: An external focus of attention enhances movement effectiveness and efficiency. , 2010 .

[8]  R. Nisbett,et al.  The influence of culture: holistic versus analytic perception , 2005, Trends in Cognitive Sciences.

[9]  Allen Newell,et al.  The psychology of human-computer interaction , 1983 .

[10]  Pedro Simões Coelho,et al.  Comparison of Likelihood and PLS Estimators for Structural Equation Modeling: A Simulation with Customer Satisfaction Data , 2010 .

[11]  Paul van Schaik,et al.  An Experimental Analysis of Experiential and Cognitive Variables in Web Navigation , 2011, Hum. Comput. Interact..

[12]  C Ray.  Some components of colour-word interference and their relationship with field dependence. , 1974, Acta psychologica.

[13]  Andy Peytchev,et al.  Self-Reports in Marketing Research: How the Questions Shape the Answers , 2007 .

[14]  Wynne W. Chin.  How to Write Up and Report PLS Analyses , 2010 .

[15]  Akira Miyake,et al.  Field dependence-independence from a working memory perspective: A dual-task investigation of the Hidden Figures Test. , 2001, Memory.

[16]  Jonathan Ling,et al.  The Psychometric Evaluation of Educational Intranets , 2005 .

[17]  J. Jaccard.  Interaction Effects in Factorial Analysis of Variance , 1997 .

[18]  Theo K. Dijkstra,et al.  Handbook of partial least squares. Concepts, methods and applications in marketing and related fields , 2008 .

[19]  Pedro Machado,et al.  Cross-cultural validation of the Educational Needs Assessment Tool in RA in 7 European countries , 2011, BMC musculoskeletal disorders.

[20]  R. Vallerand,et al.  On the Assessment of Situational Intrinsic and Extrinsic Motivation: The Situational Motivation Scale (SIMS) , 2000 .

[21]  C. Wickens,et al.  Applied Attention Theory , 2007 .

[22]  David González-Cutre,et al.  Cross-cultural invariance of the basic psychological needs in exercise scale and need satisfaction latent mean differences among Greek, Spanish, Portuguese and Turkish samples , 2013 .

[23]  Kent L. Norman,et al.  Navigational issues in the design of online self-administered questionnaires , 2001, Behav. Inf. Technol..

[24]  Retta R. Evans,et al.  Developing Valid and Reliable Online Survey Instruments Using Commercial Software Programs , 2009 .

[25]  A. Rizzo,et al.  A 3D Virtual Environment Rod and Frame Test: The Reliability and Validity of Four Traditional Scoring Methods for Older Adults , 2003, Journal of clinical and experimental neuropsychology.

[26]  Jane Webster,et al.  Perceived disorientation: an examination of a new measure to assess web design effectiveness , 2001, Interact. Comput..

[27]  Rudolf R. Sinkovics,et al.  The Use of Partial Least Squares Path Modeling in International Marketing , 2009 .

[28]  Klaus J. Templer,et al.  Internet testing: Equivalence between proctored lab and unproctored field conditions , 2008, Comput. Hum. Behav..

[29]  Joost Broekens,et al.  AffectButton: A method for reliable and valid affective self-report , 2013, Int. J. Hum. Comput. Stud..

[30]  G.P.J. Spenkelink,et al.  Visual display quality , 1996 .

[31]  Paul van Schaik,et al.  Using on-line surveys to measure three key constructs of the quality of human-computer interaction in web sites: psychometric properties and implications , 2003, Int. J. Hum. Comput. Stud..

[32]  Paul van Schaik,et al.  Design parameters of rating scales for web sites , 2007, TCHI.

[33]  R. Nisbett,et al.  Culture and systems of thought: holistic versus analytic cognition. , 2001, Psychological review.

[34]  Penelope M. Sanderson,et al.  The Effect of Individual Differences on How People Handle Interruptions , 2013 .

[35]  Paul van Schaik,et al.  The influence of font type and line length on visual search and information retrieval in web pages , 2006, Int. J. Hum. Comput. Stud..

[36]  Paul van Schaik,et al.  Five Psychometric Scales for Online Measurement of the Quality of Human-Computer Interaction in Web Sites , 2005, Int. J. Hum. Comput. Interact..

[37]  Marshall Scott Poole,et al.  Antecedents of flow in online shopping: a test of alternative models , 2009, Inf. Syst. J..

[38]  Justin Kruger,et al.  What to Do on Spring Break? , 2003, Psychological science.

[39]  David W. Moore.  Measuring New Types of Question-Order Effects: Additive and Subtractive , 2002 .

[40]  Norbert Schwarz,et al.  Assimilation and Contrast Effects in Part-Whole Question Sequences: A Conversational Logic Analysis , 1991 .

[41]  Paul van Schaik,et al.  A cognitive-experiential approach to modelling web navigation , 2012, Int. J. Hum. Comput. Stud..

[42]  Andy Peytchev,et al.  Experiments in Mobile Web Survey Design , 2010 .