Convergent and Discriminant Validity of Assessment Center Dimensions: A Conceptual and Empirical Reexamination of the Assessment Center Construct-Related Validity Paradox

This study notes that the lack of convergent and discriminant validity of assessment center ratings in the presence of content-related and criterion-related validity is paradoxical within a unitarian framework of validity. It also empirically demonstrates an application of generalizability theory to examining the convergent and discriminant validity of assessment center dimensional ratings. Generalizability analyses indicated that person, dimension, and person by dimension effects contribute large proportions of variance to the total variance in assessment center ratings. Alternately, exercise, rater, person by exercise, and dimension by exercise effects are shown to contribute little to the total variance. Correlational and confirmatory factor analyses results were consistent with the generalizability results. This provides strong evidence for the convergent and discriminant validity of the assessment center dimension ratings–a finding consistent with the conceptual underpinnings of the unitarian view of validity and inconsistent with previously reported results. Implications for future research and practice are discussed.

[1]  Charles E. Lance,et al.  Assessment Center Exercise Factors Represent Cross-Situational Specificity, Not Method Bias , 2000 .

[2]  D. Bernstein,et al.  Psychology: Fields of Application , 1998 .

[3]  F. Lievens.  Factors which Improve the Construct Validity of Assessment Centers: A Review , 1998 .

[4]  R. Hoyle.  Structural equation modeling: concepts, issues, and applications , 1997 .

[5]  A. H. Church,et al.  Managerial self-awareness in high-performing individuals in organizations. , 1997, The Journal of applied psychology.

[6]  Miguel A. Quiñones,et al.  A survey of assessment center practices in organizations in the United States. , 1997 .

[7]  Alex Howard,et al.  A reassessment of assessment centers: Challenges for the 21st century , 1997 .

[8]  Robert G. Jones,et al.  EVALUATING DEVELOPMENTAL ASSESSMENT CENTERS AS INTERVENTIONS , 1995 .

[9]  C. Russell,et al.  Two field tests of an explanation of assessment centre validity , 1995 .

[10]  Herbert W. Marsh,et al.  Latent variable models of multitrait-multimethod data. , 1995 .

[11]  David J. Woehr,et al.  Rater training for performance appraisal: A quantitative review , 1994 .

[12]  R. Brennan.  Variance Components in Generalizability Theory , 1994 .

[13]  Michael M. Harris,et al.  Does the assessment center scoring method affect the cross-situational consistency of ratings? , 1993 .

[14]  Michael M. Harris,et al.  The Measurement of Assessment Center Situations: Bem's Template Matching Technique for Examining Exercise Similarity , 1993 .

[15]  Robert G. Jones.  Construct validation of assessment center final dimension ratings: Definition and measurement issues , 1992 .

[16]  N. Schmitt,et al.  An exercise design approach to understanding assessment center dimension and exercise constructs. , 1992 .

[17]  Steven F. Cronshaw,et al.  A Process Investigation of Self-Monitoring and Leader Emergence , 1991 .

[18]  Richard J. Shavelson,et al.  Generalizability Theory: A Primer , 1991 .

[19]  Fred A. Rowe,et al.  Career guidance, career assessment, and consultancy , 1991 .

[20]  K. Kraiger,et al.  Generalizability Theory as Construct-Related Evidence of the Validity of Job Performance Ratings , 1990 .

[21]  N. Schmitt,et al.  Factors Affecting Validity of a Regionally Administered Assessment Center. , 1990 .

[22]  D. P. Baker,et al.  Construct validity of in-basket scores , 1989 .

[23]  George C. Thornton,et al.  Number of assessment center dimensions as a determinant of assessor accuracy. , 1989 .

[24]  Gerald V. Barrett,et al.  Validity of Personnel Decisions: A Conceptual Analysis of the Inferential and Evidential Bases , 1989 .

[25]  P. Sackett,et al.  A further examination of the constructs underlying assessment center ratings , 1988 .

[26]  B. Dugan.  Effects of assessor training on information use. , 1988 .

[27]  George C. Thornton,et al.  Meta-analysis of assessment center validity. , 1987 .

[28]  P. Bycio,et al.  Situational specificity in assessment center ratings: A confirmatory factor analysis. , 1987 .

[29]  Richard J. Klimoski,et al.  WHY DO ASSESSMENT CENTERS WORK? THE PUZZLE OF ASSESSMENT CENTER VALIDITY , 1987 .

[30]  Paul R. Sackett,et al.  ASSESSMENT CENTERS AND CONTENT VALIDITY: SOME NEGLECTED ISSUES , 1987 .

[31]  Frank J. Landy,et al.  Stamp collecting versus science: Validation as hypothesis testing. , 1986 .

[32]  A. Dalessio,et al.  INFLUENCE OF ASSESSMENT CENTER METHODS ON ASSESSORS’RATINGS , 1986 .

[33]  C. Russell.  Individual Decision Processes in an Assessment Center , 1985 .

[34]  Richard D. Neidig,et al.  Multiple assessment center exercises and job relatedness. , 1984 .

[35]  R. Ritchie,et al.  Assessment center correlates of women's advancement into middle management: A 7-year longitudinal analysis. , 1983 .

[36]  Janet J. Turnage,et al.  Transsituational variability in human performance within assessment centers , 1982 .

[37]  Paul R. Sackett,et al.  Constructs and assessment center dimensions: Some troubling empirical findings , 1982 .

[38]  C G Thornton,et al.  Assessment Center and managerial performance , 1982 .

[39]  and M L Tenopyr,et al.  Personnel Selection and Classification , 1982 .

[40]  H. John Bernardin,et al.  Strategies in Rater Training , 1981 .

[41]  R. Downey,et al.  Rating the ratings: Assessing the psychometric quality of rating data , 1980 .

[42]  M. Hakel,et al.  Temporal stability and individual differences in using assessment information to form overall ratings , 1979 .

[43]  N. Schmitt.  Interrater agreement in dimensionality and combination of assessment center judgments. , 1977 .

[44]  Eva Nick,et al.  The dependability of behavioral measurements: theory of generalizability for scores and profiles , 1973 .

[45]  Donald B. Rubin,et al.  The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles. , 1974 .

[46]  G. A. Miller.  THE PSYCHOLOGICAL REVIEW THE MAGICAL NUMBER SEVEN, PLUS OR MINUS TWO: SOME LIMITS ON OUR CAPACITY FOR PROCESSING INFORMATION 1 , 1956 .

[47]  R. Pintner.  Psychology applied to business. , 1921 .