An Examination of the Equivalence of Web-Based Versus Paper-and-Pencil Upward Feedback Ratings: Rater- and Ratee-Level Analyses

In this study, 5,257employees provided upward feedback ratings for 759 target managers who had the option of having their subordinates rate them using a traditional paper-and-pencil (opscan) response mode or using the company’s intranet. Preliminary analyses showed mean online ratings were more favorable than were mean paper-and-pencil ratings (d = .38). The authors then conducted rater- and ratee-level hierarchical regression analyses in which they controlled for rater and ratee characteristics. Results showed that (a) the association between response mode and upward feedback ratings was largely artifactual, with response mode accounting for 1% or less of the variance in upward feedback ratings after controlling for rater and ratee characteristics; (b) ratee characteristics accounted for much more of the variance in upward feedback ratings than did rater characteristics; and (c) response mode was unrelated to the number of missing responses.

[1]  Raj Mehta,et al.  Comparing Response Rates and Response Content in Mail versus Electronic Mail Surveys , 1995 .

[2]  Paul Rosenfeld,et al.  Responses on computer surveys: Impression management, social desirability, and the big brother syndrome , 1996 .

[3]  J. Stanton.  AN EMPIRICAL ASSESSMENT OF DATA COLLECTION USING THE INTERNET , 1998 .

[4]  Don A. Dillman,et al.  Development of a Standard E-Mail Methodology: Results of an Experiment , 1998 .

[5]  Gary J. Lautenschlager,et al.  Computer administration of questions: More desirable or more social desirability? , 1990 .

[6]  Edward W. Miles,et al.  A quasi-experimental assessment of the effect of computerizing noncognitive paper-and-pencil measurements: A test of measurement equivalence. , 1995 .

[7]  David Chan,et al.  EMPLOYEE ATTITUDE SURVEYS IN A MULTINATIONAL ORGANIZATION: CONSIDERING LANGUAGE AND CULTURE IN ASSESSING MEASUREMENT EQUIVALENCE , 1999 .

[8]  Lee Sproull,et al.  Using Electronic Mail for Data Collection in Organizational Research , 1986 .

[9]  David Antonioni,et al.  THE EFFECTS OF FEEDBACK ACCOUNTABILITY ON UPWARD APPRAISAL RATINGS , 1994 .

[10]  S. Kiesler,et al.  Response Effects in the Electronic Survey , 1986 .

[11]  Paul Rosenfeld,et al.  Impression management, social desirability, and computer administration of attitude questionnaires: Does the computer make a difference? , 1992 .

[12]  Allan H. Church,et al.  IS THERE A METHOD TO OUR MADNESS? THE IMPACT OF DATA COLLECTION METHODOLOGY ON ORGANIZATIONAL SURVEY RESULTS , 2001 .