Assessing information technology personnel: toward a behavioral rating scale

Performance ratings are used as a means to measure and record performance of individuals on any particular job. Results can then be used for promotion, increases in pay and other incentives. They, therefore, can have a large impact on a person's career and are viewed by both employers and employees as extremely important. Performance ratings vary widely and typically include a variety of measures - some of which the employee can control, and others over which the employee has no direct control. Overall, the various types of performance ratings are not perfect; each has its flaws along with its advantages. In this study, one particular type of scale - a behavioral rating scale - is proposed by taking the results of a previous study performed by Green (1989) and empirically testing a set of critical skills for use as a performance assessment instrument for information technology (IT) personnel. The instrument is provided for researchers who wish to incorporate a behavior-oriented measure in their research and for practitioners who may be looking for a behavior-oriented scale to evaluate IT personnel. The psychometric properties of the instrument as observed in this study are also presented.

[1]  Gary I. Green,et al.  Perceived Importance of Systems Analysts' Job Skills, Roles, and Non-Salary Incentives , 1989, MIS Q..

[2]  James C. Anderson,et al.  STRUCTURAL EQUATION MODELING IN PRACTICE: A REVIEW AND RECOMMENDED TWO-STEP APPROACH , 1988 .

[3]  Edward G. Carmines,et al.  Reliability and Validity Assessment , 1979 .

[4]  Leanne E. Atwater,et al.  360° feedback: Accuracy, reactions, and perceptions of usefulness. , 2001 .

[5]  H. John Bernardin,et al.  A clarification of some issues regarding the development and use of behaviorally anchored ratings scales (BARS). , 1981 .

[6]  Jack J. Baroudi,et al.  The impact of job performance evaluations on career advancement prospects: an examination of gender differences in the IS workplace , 1995 .

[7]  M. Browne,et al.  Cross-Validation Of Covariance Structures. , 1983, Multivariate behavioral research.

[8]  Elaine J. Eisenman,et al.  Effects of previous performance on evaluations of present performance. , 1985 .

[9]  Larry Hatcher,et al.  A Step-by-Step Approach to Using the SAS System for Factor Analysis and Structural Equation Modeling , 1994 .

[10]  Kathryn M. Bartol,et al.  Managing Information Systems Personnel: A Reveiw of the Literature and Managerial Implications , 1982, MIS Q..

[11]  John P. Campbell,et al.  Modeling the performance prediction problem in industrial and organizational psychology. , 1990 .

[12]  Michael J. Kavanagh.  The content issue in performance appraisal: A review. , 1971 .

[13]  Gary Klein,et al.  Performance ratings and importance of performance measures for IS staff: the different perceptions of IS users and IS staff , 2000, IEEE Trans. Engineering Management.

[14]  Elazar J. Pedhazur,et al.  Measurement, Design, and Analysis: An Integrated Approach , 1994 .

[15]  P. C. Smith,et al.  Retranslation of expectations: An approach to the construction of unambiguous anchors for rating scales. , 1963 .

[16]  Ritu Agarwal,et al.  Guided self-development: an innovative approach to management education and development applied to information systems supervisors , 1994, CPRS.

[17]  Wayne M. Wormley,et al.  Effects of Race on Organizational Experiences, Job Performance Evaluations, and Career Outcomes , 1990 .

[18]  H. Russell Bernard,et al.  Social Research Methods: Qualitative and Quantitative Approaches , 2000 .

[19]  Eileen M. Trauth,et al.  Critical Skills and Knowledge Requirements of IS Professionals: A Joint Academic/Industry Investigation , 1995, MIS Q..

[20]  Marvin D. Dunnette,et al.  The development and evaluation of behaviorally based rating scales , 1973 .

[21]  Kevin R. Murphy,et al.  Do behavioral observation scales measure observation , 1982 .

[22]  M. B. Khan,et al.  MIS professionals: Education and performance , 1990, Inf. Manag..

[23]  Robert L. Leitheiser,et al.  MIS Skills for the 1990's: A Survey of MIS Managers' Perceptions , 1992, J. Manag. Inf. Syst..

[24]  Magid Igbaria,et al.  Organizational Experiences and Career Success of MIS Professionals and Managers: An Examination of Race Differences , 1992, MIS Q..

[25]  J F Brett,et al.  360 degree feedback: accuracy, reactions, and perceptions of usefulness. , 2001, The Journal of applied psychology.

[26]  Soon Ang,et al.  Work Outcomes and Job Design for Contract Versus Permanent Information Systems Professionals on Software Development Teams , 2001, MIS Q..

[27]  C. Heckler.  A Step-by-Step Approach to Using the SAS™ System for Factor Analysis and Structural Equation Modeling , 1996 .

[28]  A. Bedeian,et al.  Job-related perceptions of male and female government, industrial, and public accountants , 1984 .

[29]  Robert D. Smith,et al.  Placement, performance and turnover of information systems professionals: implications for HRIS , 1993, SIGCPR '93.

[30]  Magid Igbaria,et al.  The Career Advancement Prospects of Managers and Professionals: Are MIS Employees Unique? , 1992 .

[31]  Thomas W. Ferratt,et al.  Are Information Systems People Different? An Investigation of How They Are and Should Be Managed , 1988, MIS Q..

[32]  M. Browne,et al.  Single Sample Cross-Validation Indices for Covariance Structures. , 1989, Multivariate behavioral research.

[33]  Richard D. Arvey,et al.  A Guttman approach to the development of behaviorally based rating scales for systems analysts and programmer/analysts. , 1974 .

[34]  Kevin R. Murphy,et al.  Behavioral anchors as a source of bias in rating , 1987 .

[35]  Luis R. Gomez-Mejia,et al.  Evaluating employee performance: Does the appraisal instrument make a difference? , 1988 .

[36]  Andrew Neal,et al.  Technology and performance , 1999 .

[37]  Gary P. Latham,et al.  BEHAVIORAL OBSERVATION SCALES FOR PERFORMANCE APPRAISAL PURPOSES , 1977 .

[38]  Karl G. Jöreskog,et al.  Lisrel 8: Structural Equation Modeling With the Simplis Command Language , 1993 .

[39]  J Noak,et al.  Performance rating. , 1999, Nursing standard (Royal College of Nursing (Great Britain) : 1987).

[40]  Ronald H. Rasch,et al.  Factors Affecting Software Developers' Performance: An Integrated Approach , 1992, MIS Q..

[41]  Aharon Tziner,et al.  A Comparison of Three Methods of Performance Appraisal with Regard to Goal Properties, Goal Perception, and Ratee Satisfaction , 2000 .

[42]  Craig Eric Schneier,et al.  AN EMPIRICAL INVESTIGATION OF PERCEPTIONS OF RATEE BEHAVIOR FREQUENCY AND RATEE BEHAVIOR CHANGE USING BEHAVIORAL EXPECTATION SCALES (BES) , 1977 .

[43]  Nicholas P. Vitalari,et al.  Knowledge as a Basis for Expertise in Systems Analysis: An Empirical Study , 1985, MIS Q..

[44]  Brenda L. Killingsworth,et al.  A Model for Motivating and Measuring Quality Performance in Information Systems Staff , 2001, Inf. Syst. Manag..

[45]  Gary P. Latham,et al.  The Practicality of Behavioral Observation Scales, Behavioral Expectation Scales, and Trait Scales. , 1986 .

[46]  Gary P. Latham,et al.  Dutch Reactions to Behavioral Observation, Behavioral Expectation, and Trait Scales , 1995 .

[47]  James D. McKeen,et al.  The Evolution of IS Job Skills: A Content Analysis of IS Job Advertisements from 1970 to 1990 , 1995, MIS Q..

[48]  Richard D. Arvey,et al.  Fairness in the selection of employees , 1992 .

[49]  D. Harris.  Organizational Linkages: Understanding the Productivity Paradox, , 1997 .

[50]  M. Browne,et al.  Alternative Ways of Assessing Model Fit , 1992 .