Revised estimates of dimension and exercise variance components in assessment center postexercise dimension ratings.

The authors reanalyzed assessment center (AC) multitrait-multimethod (MTMM) matrices containing correlations among postexercise dimension ratings (PEDRs) reported by F. Lievens and J. M. Conway (2001). Unlike F. Lievens and J. M. Conway, who used a correlated dimension-correlated uniqueness model, we used a different set of confirmatory-factor-analysis-based models (1-dimension-correlated Exercise and 1-dimension-correlated uniqueness models) to estimate dimension and exercise variance components in AC PEDRs. Results of reanalyses suggest that, consistent with previous narrative reviews, exercise variance components dominate over dimension variance components after all. Implications for AC construct validity and possible redirections of research on the validity of ACs are discussed.

[1]  Lawrence R. James,et al.  A Note on Validity Generalization Procedures. , 1986 .

[2]  P. Bentler,et al.  Cutoff criteria for fit indexes in covariance structure analysis : Conventional criteria versus new alternatives , 1999 .

[3]  Alene S. Becker.  The effects of a reduction in assessor roles on the convergent and discriminant validity of assessment center ratings , 1990 .

[4]  J. Conway,et al.  Dimension and exercise variance in assessment center scores: a large-scale evaluation of multitrait-multimethod studies. , 2001, The Journal of applied psychology.

[5]  C. Lance,et al.  Assessor cognitive processes in an operational assessment center. , 2004, The Journal of applied psychology.

[6]  Winfred Arthur,et al.  The Construct-Related Validity of Assessment Center Ratings: A Review and Meta-Analysis of the Role of Methodological Factors , 2003 .

[7]  F. Schmidt,et al.  The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. , 1998 .

[8]  F. Lievens.  Factors which Improve the Construct Validity of Assessment Centers: A Review , 1998 .

[9]  Charles E. Lance,et al.  Assessment Center Exercise Factors Represent Cross-Situational Specificity, Not Method Bias , 2000 .

[10]  Ivan T. Robertson,et al.  The psychometric properties and design of managerial assessment centres: Dimensions into exercises won't go , 1987 .

[11]  Mark S. Teachout,et al.  AIR FORCE RESEARCH TO LINK STANDARDS FOR ENLISTMENT TO ON-THE-JOB PERFORMANCE , 1991 .

[12]  Phillip E. Lowry.  The Assessment Center Process: Assessing Leadership in the Public Sector , 1995 .

[13]  Task Force on Assessment Center Guidelines.  Guidelines and ethical considerations for Assessment Center operations , 1989 .

[14]  D. B. Bell,et al.  Differences in Spouses' Perceptions of Family Adaptation , 1997 .

[15]  H. G. Osburn,et al.  Effects of the Rating Process on the Construct Validity of Assessment Center Dimension Evaluations , 2000 .

[16]  L. Newman,et al.  When the Stakes are Higher: Self-esteem Instability and Self-handicapping , 1997 .

[17]  George C. Thornton,et al.  Meta-analysis of assessment center validity. , 1987 .

[18]  Deidra J. Schleicher,et al.  A new frame for frame-of-reference training: enhancing the construct validity of assessment centers. , 2002, The Journal of applied psychology.

[19]  Bert F. Green,et al.  Performance assessment for the workplace , 1991 .

[20]  G. Dobbins,et al.  New Evidence On the Construct Validity of Diagnostic Assessment Centers: The Findings May Not Be So Troubling After All , 1997 .

[21]  O. Köller,et al.  Der Einfluss unterschiedlicher Auswertungsmethoden auf die Konstruktvalidität von Assessment-Centern , 1994 .

[22]  D. A. Kenny,et al.  Analysis of the multitrait^multimethod matrix by confirmatory factor analysis , 1992 .

[23]  Nanja J. Kolk,et al.  The Transparent Assessment Centre: The Effects of Revealing Dimensions to Candidates , 2003 .

[24]  M. Browne,et al.  The decomposition of multitrait-multimethod matrices. , 1984, The British journal of mathematical and statistical psychology.

[25]  F. Lievens.  Assessors and use of assessment centre dimensions: a fresh look at a troubling issue* , 2001 .

[26]  Alex Howard,et al.  A reassessment of assessment centers: Challenges for the 21st century , 1997 .

[27]  C. Cooper,et al.  International review of industrial and organizational psychology , 1986 .

[28]  H. Marsh,et al.  Confirmatory Factor Analyses of Multitrait-Multimethod Data: A Comparison of Alternative Models , 1991 .

[29]  H. Marsh.  Confirmatory factor analysis models of factorial invariance: A multifaceted approach , 1994 .

[30]  C. Lance,et al.  Good News: Work Sample Administrators' Global Performance Judgments are (About) as Valid as We've Suspected , 2000 .

[31]  Gordon W. Cheung,et al.  The Effects of Model Parsimony and Sampling Error on the Fit of Structural Equation Models , 2001 .

[32]  Winfred Arthur,et al.  Convergent and Discriminant Validity of Assessment Center Dimensions: A Conceptual and Empirical Reexamination of the Assessment Center Construct-Related Validity Paradox , 2000 .

[33]  Warren S. Bobrow,et al.  Development and validation of an assessment center during organizational change. , 1997 .

[34]  M. London.  How people evaluate others in organizations , 2001 .

[35]  Samuel B. Pond,et al.  MANAGERIAL FUNCTIONS: AN ALTERNATIVE TO TRADITIONAL ASSESSMENT CENTER DIMENSIONS? , 1994 .

[36]  P. Bentler,et al.  Comparative fit indexes in structural models. , 1990, Psychological bulletin.

[37]  Bernhard Wilpert,et al.  Applied psychology : an international review , 1989 .

[38]  D. Truxillo,et al.  Assessment center construct validity and behavioral checklists: Some additional findings. , 1997 .

[39]  F. Lievens.  Trying to understand the different pieces of the construct validity puzzle of assessment centers: an examination of assessor and assessee effects. , 2002, Journal of Applied Psychology.

[40]  Paul R. Sackett,et al.  Constructs and assessment center dimensions: Some troubling empirical findings , 1982 .

[41]  F. Lievens,et al.  Understanding the assessment center process: Where are we now? , 2001 .

[42]  A. Fredricks.  Assessment center ratings: Models and process , 1989 .

[43]  A. Sagie,et al.  Assessor type, number of distinguishable dimension categories, and assessment centre construct validity , 1997 .

[44]  M. Kleinmann,et al.  Transparency: A Necessary Requirement for the Construct Validity of Assessment Centres , 1996 .

[45]  S. Scullen.  Using Confirmatory Factor Analysis of Correlated Uniquenesses to Estimate Method Variance in Multitrait-Multimethod Matrices , 1999 .

[46]  Lawrence R. James,et al.  The unmeasured variables problem in path analysis. , 1980 .

[47]  Paul E. Spector,et al.  Estimation Problems in the Block-Diagonal Model of the Multitrait-Multimethod Matrix , 1990 .

[48]  Steven E. Scullen,et al.  Bias in the Correlated Uniqueness Model for MTMM Data , 2004 .

[49]  P. Bycio,et al.  Situational specificity in assessment center ratings: A confirmatory factor analysis. , 1987 .

[50]  N. Schmitt,et al.  An exercise design approach to understanding assessment center dimension and exercise constructs. , 1992 .

[51]  John W. Fleenor.  Constructs and developmental assessment centers: Further troubling empirical findings , 1996 .

[52]  H. Marsh.  Confirmatory Factor Analyses of Multitrait-Multimethod Data: Many Problems and a Few Solutions , 1989 .

[53]  Michael M. Harris,et al.  Does the assessment center scoring method affect the cross-situational consistency of ratings? , 1993 .

[54]  Steven E. Scullen,et al.  A critique of the correlated trait-correlated method and correlated uniqueness models for multitrait-multimethod data. , 2002, Psychological methods.