A meta-analytic evaluation of the impact of dimension and exercise factors on assessment center ratings.

Two recent reviews have attempted to summarize findings quantitatively regarding assessment center (AC) construct-related validity (i.e., Lance, Lambert, Gewin, Lievens, & Conway, 2004; Lievens & Conway, 2001). Unlike these previous studies, which reanalyzed individual multitrait-multimethod (MTMM) matrices from previously published research, the authors recoded and combined past matrices into a single MTMM matrix. This matrix, comprised of 6 dimensions each measured by 6 exercises, was then analyzed, providing a more generalizable set of results. Both dimensions and exercises were found to contribute substantially to AC ratings. Specific dimensions (i.e., communication, influencing others, organizing and planning, and problem solving) appeared more construct valid than others (i.e., consideration/awareness of others and drive). Implications for AC design and practice are discussed.

[1]  N. Schmitt,et al.  An exercise design approach to understanding assessment center dimension and exercise constructs. , 1992 .

[2]  A. Fredricks.  Assessment center ratings: Models and process , 1989 .

[3]  Deidra J. Schleicher,et al.  A new frame for frame-of-reference training: enhancing the construct validity of assessment centers. , 2002, The Journal of applied psychology.

[4]  M. Kleinmann,et al.  Construct validity of assessment centers: Appropriate use of confirmatory factor analysis and suitable construction principles. , 1997 .

[5]  F. Lievens.  Assessors and use of assessment centre dimensions: a fresh look at a troubling issue* , 2001 .

[6]  M. Browne,et al.  Single Sample Cross-Validation Indices for Covariance Structures. , 1989, Multivariate behavioral research.

[7]  Herbert W. Marsh,et al.  Latent variable models of multitrait-multimethod data. , 1995 .

[8]  Winfred Arthur,et al.  Convergent and Discriminant Validity of Assessment Center Dimensions: A Conceptual and Empirical Reexamination of the Assessment Center Construct-Related Validity Paradox , 2000 .

[9]  George C. Thornton,et al.  Construct Validity of Self- and Peer Evaluations of Performance Dimensions in an Assessment Center , 1992 .

[10]  Janet J. Turnage,et al.  Transsituational variability in human performance within assessment centers , 1982 .

[11]  O. Köller,et al.  Der Einfluss unterschiedlicher Auswertungsmethoden auf die Konstruktvalidität von Assessment-Centern , 1994 .

[12]  D. P. Baker,et al.  Construct validity of in-basket scores , 1989 .

[13]  F. Lievens.  Factors which Improve the Construct Validity of Assessment Centers: A Review , 1998 .

[14]  J. H. Steiger.  Structural Model Evaluation and Modification: An Interval Estimation Approach. , 1990, Multivariate behavioral research.

[15]  Bernhard Wilpert,et al.  Applied psychology : an international review , 1989 .

[16]  L A Hayduk.  Should Model Modifications be Oriented Toward Improving Data Fit or Encouraging Creative and Analytical Thinking? , 1990, Multivariate behavioral research.

[17]  Filip Lievens,et al.  Revised estimates of dimension and exercise variance components in assessment center postexercise dimension ratings. , 2004, The Journal of applied psychology.

[18]  R. Taft.  Multiple methods of personality assessment. , 1959, Psychological bulletin.

[19]  John W. Fleenor.  Constructs and developmental assessment centers: Further troubling empirical findings , 1996 .

[20]  H. Marsh.  Confirmatory Factor Analyses of Multitrait-Multimethod Data: Many Problems and a Few Solutions , 1989 .

[21]  Samuel B. Pond,et al.  MANAGERIAL FUNCTIONS: AN ALTERNATIVE TO TRADITIONAL ASSESSMENT CENTER DIMENSIONS? , 1994 .

[22]  Michael M. Harris,et al.  The Measurement of Assessment Center Situations: Bem's Template Matching Technique for Examining Exercise Similarity , 1993 .

[23]  R. Hoyle.  Structural equation modeling: concepts, issues, and applications , 1997 .

[24]  John E. Hunter,et al.  Methods of Meta-Analysis: Correcting Error and Bias in Research Findings , 1991 .

[25]  H. Marsh,et al.  Confirmatory Factor Analyses of Multitrait-Multimethod Data: A Comparison of Alternative Models , 1991 .

[26]  Alex Howard,et al.  A reassessment of assessment centers: Challenges for the 21st century , 1997 .

[27]  L. Shore,et al.  CONSTRUCT VALIDITY OF TWO CATEGORIES OF ASSESSMENT CENTER DIMENSION RATINGS , 1990 .

[28]  D. Bray,et al.  The assessment center in the measurement of potential for business management. , 1966, Psychological monographs.

[29]  Alene S. Becker.  The effects of a reduction in assessor roles on the convergent and discriminant validity of assessment center ratings , 1990 .

[30]  C. Lance,et al.  Assessor cognitive processes in an operational assessment center. , 2004, The Journal of applied psychology.

[31]  Miguel A. Quiñones,et al.  A survey of assessment center practices in organizations in the United States. , 1997 .

[32]  Ann Howard,et al.  An Assessment of Assessment Centers , 1974 .

[33]  Eric Anthony Day,et al.  A META‐ANALYSIS OF THE CRITERION‐RELATED VALIDITY OF ASSESSMENT CENTER DIMENSIONS , 2003 .

[34]  G. Dobbins,et al.  New Evidence On the Construct Validity of Diagnostic Assessment Centers: The Findings May Not Be So Troubling After All , 1997 .

[35]  Winfred Arthur,et al.  The Construct-Related Validity of Assessment Center Ratings: A Review and Meta-Analysis of the Role of Methodological Factors , 2003 .

[36]  Charles E. Lance,et al.  Assessment Center Exercise Factors Represent Cross-Situational Specificity, Not Method Bias , 2000 .

[37]  J. Conway,et al.  Dimension and exercise variance in assessment center scores: a large-scale evaluation of multitrait-multimethod studies. , 2001, The Journal of applied psychology.

[38]  H. G. Osburn,et al.  Effects of the Rating Process on the Construct Validity of Assessment Center Dimension Evaluations , 2000 .

[39]  M. Born,et al.  Impact of Common Rater Variance on Construct Validity of Assessment Center Dimension Judgments , 2002 .

[40]  P. Bycio,et al.  Situational specificity in assessment center ratings: A confirmatory factor analysis. , 1987 .

[41]  A. Sagie,et al.  Assessor type, number of distinguishable dimension categories, and assessment centre construct validity , 1997 .

[42]  M. Kleinmann,et al.  Transparency: A Necessary Requirement for the Construct Validity of Assessment Centres , 1996 .

[43]  S. Scullen.  Using Confirmatory Factor Analysis of Correlated Uniquenesses to Estimate Method Variance in Multitrait-Multimethod Matrices , 1999 .