Measurement Equivalence of Paper‐and‐Pencil and Internet Organisational Surveys: A Large Scale Examination in 16 Countries

In multinational surveys, mixed-mode administration modes (e.g. combining Internet and paper-and-pencil administration) are increasingly used. To date, no studies have investigated whether measurement equivalence exists between Internet data collection and data collection using the conventional paper-and-pencil method in organisational surveys which include a large number of countries. This paper examined the measurement equivalence of a truly global organisational survey across Internet and paper-and-pencil survey administrations. Data from an organisational survey in 16 countries (N = 52,461) across the globe were used to assess the measurement equivalence of an organisational climate measure within each country in which the survey was administered. The empirical data provided strong indications which support the measurement equivalence of the multi-item survey instrument across Internet and paper-and-pencil surveys in virtually all countries in which the survey was conducted. These findings suggest that merging data obtained through Internet and paper-and-pencil data administration in a particular country is legitimate as no evidence was found for differential effects across both modes of data collection.

[1]  A. Bowling.  Mode of questionnaire administration can have serious effects on data quality. , 2005, Journal of public health.

[2]  Bruce Thompson,et al.  Score Reliability in Webor Internet-Based Surveys: Unnumbered Graphic Rating Scales versus Likert-Type Scales , 2001 .

[3]  R. Vandenberg.  Toward a Further Understanding of and Improvement in Measurement Invariance Methods and Procedures , 2002 .

[4]  R. K. Biswas.  Organisational Climate and Culture , 2010 .

[5]  Jeffrey M. Stanton,et al.  Using Internet/Intranet Web Pages to Collect Organizational Research Data , 2001 .

[6]  Y. Poortinga,et al.  Response Styles in Rating Scales , 2004 .

[7]  L. J. Williams,et al.  Exploratory and confirmatory factor analysis: guidelines, issues, and alternatives , 1997 .

[8]  E. Surface,et al.  FROM PAPER TO PIXELS: MOVING PERSONNEL SURVEYS TO THE WEB , 2003 .

[9]  P. Bentler,et al.  Comparative fit indexes in structural models. , 1990, Psychological bulletin.

[10]  D. Gerbing,et al.  Viability of exploratory factor analysis as a precursor to confirmatory factor analysis , 1996 .

[11]  Gi Woong Yun,et al.  Comparative Response to a Survey Executed by Post, E-mail, & Web Form , 2006, J. Comput. Mediat. Commun..

[12]  Allen I. Kraut,et al.  Organizational surveys : tools for assessment and change , 1996 .

[13]  P. Bentler,et al.  Significance Tests and Goodness of Fit in the Analysis of Covariance Structures , 1980 .

[14]  Jaeyool Boo,et al.  Computerized and Paper-and-Pencil Versions of the Rosenberg Self-Esteem Scale: A Comparison of Psychometric Features and Respondent Preferences , 2001 .

[15]  Allan H. Church,et al.  Designing and using organizational surveys : a seven-step process , 2001 .

[16]  Lisa L. Roberts,et al.  Effects of Data Collection Method on Organizational Climate Survey Results , 2004 .

[17]  Robert J. Vandenberg,et al.  A Central Question in Cross-Cultural Research: Do Employees of Different Cultures Interpret Work-related Measures in an Equivalent Manner?: , 1994 .

[18]  Ulf-Dietrich Reips.  Chapter 4 – The Web Experiment Method: Advantages, Disadvantages, and Solutions , 2000 .

[19]  Michael H. Birnbaum,et al.  Introduction to Behavioral Research on the Internet , 2000 .

[20]  D. Dillman.  Mail and internet surveys: The tailored design method, 2nd ed. , 2007 .

[21]  Arthur G. Bedeian,et al.  The Measurement Equivalence of Web-Based and Paper-and-Pencil Measures of Transformational Leadership , 2006 .

[22]  Bert Weijters.  Response styles in consumer research , 2006 .

[23]  T. Little.  Mean and Covariance Structures (MACS) Analyses of Cross-Cultural Data: Practical and Theoretical Issues. , 1997, Multivariate behavioral research.

[24]  Peter M. Bentler,et al.  EQS : structural equations program manual , 1989 .

[25]  Robert E. Ployhart,et al.  WEB‐BASED AND PAPER‐AND‐PENCIL TESTING OF APPLICANTS IN A PROCTORED SETTING: ARE PERSONALITY, BIODATA, AND SITUATIONAL JUDGMENT TESTS COMPARABLE? , 2003 .

[26]  Gary J. Lautenschlager,et al.  Computer administration of questions: More desirable or more social desirability? , 1990 .

[27]  Lee Sproull,et al.  Using Electronic Mail for Data Collection in Organizational Research , 1986 .

[28]  Don A. Dillman,et al.  Development of a Standard E-Mail Methodology: Results of an Experiment , 1998 .

[29]  Young Ik Cho,et al.  The Relation Between Culture and Response Styles , 2005 .

[30]  Zeki Simsek,et al.  A Primer on Internet Organizational Surveys , 2001 .

[31]  D. Chan,et al.  Detection of Differential Item Functioning on the Kirton Adaption-Innovation Inventory Using Multiple-Group Mean and Covariance Structure Analyses , 2000, Multivariate behavioral research.

[32]  J. Epstein,et al.  Insuring sample equivalence across internet and paper-and-pencil assessments , 2001, Comput. Hum. Behav..

[33]  P. Bentler,et al.  Cutoff criteria for fit indexes in covariance structure analysis : Conventional criteria versus new alternatives , 1999 .

[34]  Nicholas W. Jankowski,et al.  Conducting Online Surveys , 2006 .

[35]  Fiona Stewart,et al.  Internet Communication and Qualitative Research , 2000 .

[36]  P. Ferrando,et al.  IRT-related factor analytic procedures for testing the equivalence of paper-and-pencil and Internet-administered questionnaires. , 2005, Psychological methods.

[37]  F. Drasgow.  Scrutinizing psychological tests: Measurement equivalence and equivalent relations with external variables are the central issues , 1984 .

[38]  Dag Sörbom,et al.  An alternative to the methodology for analysis of covariance , 1978 .

[39]  Michael T. Brannick,et al.  Critical comments on applying covariance structure modeling , 1995 .

[40]  David Chan,et al.  EMPLOYEE ATTITUDE SURVEYS IN A MULTINATIONAL ORGANIZATION: CONSIDERING LANGUAGE AND CULTURE IN ASSESSING MEASUREMENT EQUIVALENCE , 1999 .

[41]  W. Meredith.  Measurement invariance, factor analysis and factorial invariance , 1993 .

[42]  Paul Rosenfeld,et al.  Impression management, social desirability, and computer administration of attitude questionnaires: Does the computer make a difference? , 1992 .

[43]  R. Vandenberg,et al.  A Review and Synthesis of the Measurement Invariance Literature: Suggestions, Practices, and Recommendations for Organizational Research , 2000 .

[44]  Lawrence R. James,et al.  Organizational climate: A review of theory and research. , 1974 .

[45]  Xianggui Qu,et al.  Multivariate Data Analysis , 2007, Technometrics.

[46]  Gordon W. Cheung,et al.  Evaluating Goodness-of-Fit Indexes for Testing Measurement Invariance , 2002 .

[47]  E. Kevin Kelloway,et al.  Structural equation modelling in perspective. , 1995 .

[48]  J. H. Steiger.  Structural Model Evaluation and Modification: An Interval Estimation Approach. , 1990, Multivariate behavioral research.

[49]  J. Stanton.  AN EMPIRICAL ASSESSMENT OF DATA COLLECTION USING THE INTERNET , 1998 .

[50]  Christopher P. Parker,et al.  Relationships between psychological climate perceptions and work outcomes: a meta-analytic review , 2003 .

[51]  Kevin R. Murphy,et al.  Individual differences and behavior in organizations , 1996 .

[52]  D. Sörbom.  A GENERAL METHOD FOR STUDYING DIFFERENCES IN FACTOR MEANS AND FACTOR STRUCTURE BETWEEN GROUPS , 1974 .

[53]  Cheri Ostroff.  The Effects of Climate and Personal Influences on Individual Behavior and Attitudes in Organizations , 1993 .

[54]  Fiona Stewart,et al.  Internet Communication and Qualitative Research: A Handbook for Researching Online , 2000 .

[55]  D. Magnusson.  Toward A Psychology of Situations : An Interactional Perspective , 1981 .

[56]  John L. Smith,et al.  Using the Internet for psychological research: personality testing on the World Wide Web. , 1999, British journal of psychology.

[57]  Fritz Drasgow,et al.  Study of the measurement bias of two standardized psychological tests. , 1987 .