Alternate Approaches to Understanding the Psychometric Properties of Assessment Centers: An Analysis of the Structure and Equivalence of Exercise Ratings

The present study uses an alternate analytical framework to examine the degree to which performance is differentiated by dimensions in assessment center (AC) exercises and whether these performance dimensions are rated on the same scale across exercises. Confirmatory factor analysis likelihood ratio tests supported the presence of three broad latent performance dimensions in each of three AC exercises. Additional tests revealed that five of six manifest performance dimensions were rated on the same psychological scale across exercises. Taken together, our results support a multidimensional interpretation of AC exercises and provide empirical support to the notion that differences in AC performance across exercises reflect true performance, rather than a measurement artifact.

[1]  S. Messick.  Validity of Psychological Assessment: Validation of Inferences from Persons' Responses and Performances as Scientific Inquiry into Score Meaning. Research Report RR-94-45. , 1994 .

[2]  John P. Campbell,et al.  MODELING JOB PERFORMANCE IN A POPULATION OF JOBS , 1990 .

[3]  L. Shore,et al.  CONSTRUCT VALIDITY OF TWO CATEGORIES OF ASSESSMENT CENTER DIMENSION RATINGS , 1990 .

[4]  Chockalingam Viswesvaran,et al.  Is there a general factor in ratings of job performance? A meta-analytic framework for disentangling substantive and error influences. , 2005, The Journal of applied psychology.

[5]  D. Ones,et al.  Assessment Center Dimensions: Individual Differences Correlates and Meta-Analytic Incremental Validity , 2009 .

[6]  Steven D. Norton,et al.  The Empirical and Content Validity of Assessment Centers vs. Traditional Methods for Predicting Managerial Success , 1977 .

[7]  Winfred Arthur,et al.  Convergent and Discriminant Validity of Assessment Center Dimensions: A Conceptual and Empirical Reexamination of the Assessment Center Construct-Related Validity Paradox , 2000 .

[8]  Nanja J. Kolk,et al.  A Triadic Approach to the Construct Validity of the Assessment Center: The Effect of Categorizing Di , 2004 .

[9]  Charles E. Lance,et al.  RATER SOURCE EFFECTS ARE ALIVE AND WELL AFTER ALL , 2010 .

[10]  G. C. Thornton,et al.  The Construct Validity of the Assessment Center Method and Usefulness of Dimensions as Focal Constructs , 2008, Industrial and Organizational Psychology.

[11]  C. Lance,et al.  Assessor cognitive processes in an operational assessment center. , 2004, The Journal of applied psychology.

[12]  Ralph M. Stogdill,et al.  Manual for the leader behavior description questionnaire - Form XII , 1963 .

[13]  Mark A. Wilson,et al.  Factors affecting the consensus judgment process in managerial assessment centers. , 1982 .

[14]  J. Stillman,et al.  Rating Tasks Versus Dimensions in Assessment Centers: A Psychometric Comparison , 2005 .

[15]  Winfred Arthur,et al.  The Construct-Related Validity of Assessment Center Ratings: A Review and Meta-Analysis of the Role of Methodological Factors , 2003 .

[16]  Brian J. Hoffman.  Disentangling the Meaning of Multisource Feedback: An Examination of the Nomological Network Surrounding Source and Dimension Factors , 2006 .

[17]  Charles E. Lance,et al.  Assessment Center Exercise Factors Represent Cross-Situational Specificity, Not Method Bias , 2000 .

[18]  Ronald F. Piccolo,et al.  Transformational and transactional leadership: a meta-analytic test of their relative validity. , 2004, The Journal of applied psychology.

[19]  John P. Meriac,et al.  Expanding the criterion domain? A quantitative review of the OCB literature. , 2007, The Journal of applied psychology.

[20]  F. Lievens.  Trying to understand the different pieces of the construct validity puzzle of assessment centers: an examination of assessor and assessee effects. , 2002, The Journal of applied psychology.

[21]  Michael A. McDaniel,et al.  Can Basic Individual Differences Shed Light on the Construct Meaning of Assessment Center Evaluations , 2003 .

[22]  J. H. Steiger.  Structural Model Evaluation and Modification: An Interval Estimation Approach. , 1990, Multivariate behavioral research.

[23]  Paul R. Sackett,et al.  Constructs and assessment center dimensions: Some troubling empirical findings , 1982 .

[24]  J. Pellegrino,et al.  Testing: Theoretical and Applied Perspectives , 1989 .

[25]  Mark C. Bowler,et al.  A meta-analytic evaluation of the impact of dimension and exercise factors on assessment center ratings. , 2006, The Journal of applied psychology.

[26]  S. Messick.  Validity of Psychological Assessment: Validation of Inferences from Persons' Responses and Performances as Scientific Inquiry into Score Meaning. Research Report RR-94-45. , 1994 .

[27]  Gordon W. Cheung,et al.  Evaluating Goodness-of-Fit Indexes for Testing Measurement Invariance , 2002 .

[28]  Chockalingam Viswesvaran,et al.  The moderating influence of job performance dimensions on convergence of supervisory and peer ratings of job performance: unconfounding construct-level convergence and rating difficulty. , 2002, The Journal of applied psychology.

[29]  K. Widaman,et al.  Multitrait-multimethod models in aging research. , 1992, Experimental aging research.

[30]  Cynthia D. McCauley,et al.  BENCHMARKS: An instrument for diagnosing managerial strengths and weaknesses. , 1990 .

[31]  M. Born,et al.  A construct-driven investigation of gender differences in a leadership-role assessment center. , 2006, The Journal of applied psychology.

[32]  R. Vandenberg,et al.  A Review and Synthesis of the Measurement Invariance Literature: Suggestions, Practices, and Recommendations for Organizational Research , 2000 .

[33]  J. Conway,et al.  Distinguishing contextual performance from task performance for managerial jobs. , 1999 .

[34]  J. Salgado,et al.  Test–Retest Reliability of Ratings of Job Performance Dimensions in Managers , 2003 .

[35]  Eric Anthony Day,et al.  A META‐ANALYSIS OF THE CRITERION‐RELATED VALIDITY OF ASSESSMENT CENTER DIMENSIONS , 2003 .

[36]  F. Lievens.  What Does Exercise-Based Assessment Really Mean? , 2008, Industrial and Organizational Psychology.

[37]  J. Conway,et al.  Analysis and Design of Multitrait-Multirater Performance Appraisal Studies , 1996 .

[38]  D. Bray.  The assessment center and the study of lives. , 1982 .

[39]  Matthew S Fleisher,et al.  Further evidence for the validity of assessment center dimensions: a meta-analysis of the incremental criterion-related validity of dimension ratings. , 2008, The Journal of applied psychology.

[40]  Kevin R. Murphy,et al.  Dimensions of Job Performance , 1988 .

[41]  C. A. Blair,et al.  EXERCISES AND DIMENSIONS ARE THE CURRENCY OF ASSESSMENT CENTERS , 2011 .

[42]  Paul E. Spector,et al.  Estimation Problems in the Block-Diagonal Model of the Multitrait-Multimethod Matrix , 1990 .

[43]  Filip Lievens,et al.  A closer look at the frame-of-reference effect in personality scale scores and validity. , 2008, The Journal of applied psychology.

[44]  Walter C. Borman,et al.  More Progress Toward a Taxonomy of Managerial Performance Requirements , 1993 .

[45]  D. Ones,et al.  A Pragmatic View of Assessment Center Exercises and Dimensions , 2008, Industrial and Organizational Psychology.

[46]  Elazar J. Pedhazur,et al.  Measurement, Design, and Analysis: An Integrated Approach , 1994 .

[47]  J. Diefendorff,et al.  Measurement Equivalence and Multisource Ratings for Non-managerial Positions: Recommendations for Research and Practice , 2005 .

[48]  J. Moses.  Assessment Centers Work, but for Different Reasons , 2008, Industrial and Organizational Psychology.

[49]  F. L. Wells,et al.  A Statistical Study of Literary Merit with Remarks on Some New Phases of the Method , 2009 .

[50]  Jack M. Feldman,et al.  Beyond Attribution Theory: Cognitive Processes in Performance Appraisal , 1981 .

[51]  George C. Thornton,et al.  Meta-analysis of assessment center validity. , 1987 .

[52]  P. Bentler,et al.  Comparative fit indexes in structural models. , 1990, Psychological bulletin.

[53]  Phillip W. Braddy,et al.  Power and sensitivity of alternative fit indices in tests of measurement invariance. , 2008, The Journal of applied psychology.

[54]  Filip Lievens,et al.  Revised estimates of dimension and exercise variance components in assessment center postexercise dimension ratings. , 2004, The Journal of applied psychology.

[55]  Chockalingam Viswesvaran,et al.  Comparative analysis of the reliability of job performance ratings , 1996 .

[56]  Paul R. Sackett,et al.  ASSESSMENT CENTERS AND CONTENT VALIDITY: SOME NEGLECTED ISSUES , 1987 .

[57]  N. Schmitt.  Interrater agreement in dimensionality and combination of assessment center judgments. , 1977 .

[58]  E. Thorndike.  A constant error in psychological ratings. , 1920 .

[59]  Filip Lievens,et al.  Large-scale investigation of the role of trait activation theory for understanding assessment center convergent and discriminant validity. , 2006, The Journal of applied psychology.

[60]  Ann Howard,et al.  Making Assessment Centers Work the Way They Are Supposed To , 2008, Industrial and Organizational Psychology.

[61]  J. Stillman,et al.  Task-Based Assessment Centers: Empirical Support for a Systems Model , 2010 .

[62]  Charles E. Lance,et al.  Why Assessment Centers Do Not Work the Way They Are Supposed To , 2008, Industrial and Organizational Psychology.

[63]  H. Marsh.  Confirmatory Factor Analyses of Multitrait-Multimethod Data: Many Problems and a Few Solutions , 1989 .

[64]  Nambury S. Raju,et al.  Peer and subordinate performance appraisal measurement equivalence. , 1998 .

[65]  Bryan Dawson,et al.  Method Effects, Measurement Error, and Substantive Conclusions , 2010 .

[66]  Steven E. Scullen,et al.  Evidence of the construct validity of developmental ratings of managerial performance. , 2003, The Journal of applied psychology.

[67]  Adam W. Meade,et al.  A Monte Carlo Investigation of Assessment Center Construct Validity Models , 2007 .

[68]  C. Russell,et al.  Two field tests of an explanation of assessment centre validity , 1995 .

[69]  Robert E. Wood,et al.  Self- versus others' ratings as predictors of assessment center ratings: Validation evidence for 360-degree feedback programs. , 2002 .

[70]  Brian J. Hoffman,et al.  DISENTANGLING THE MEANING OF MULTISOURCE PERFORMANCE RATING SOURCE AND DIMENSION FACTORS , 2009 .

[71]  A. Furnham.  Mentoring at work , 2012 .

[72]  Eric Anthony Day,et al.  Mend It, Don’t End It: An Alternate View of Assessment Center Construct-Related Validity Evidence , 2008, Industrial and Organizational Psychology.

[73]  M. Brannick.  Back to Basics of Test Construction and Scoring , 2008, Industrial and Organizational Psychology.

[74]  Sarah A. Hezlett,et al.  TRAIT, RATER AND LEVEL EFFECTS IN 360‐DEGREE PERFORMANCE RATINGS , 1998 .

[75]  J. Guilford.  Psychometric methods, 2nd ed. , 1954 .

[76]  Charles E. Lance,et al.  Where Have We Been, How Did We Get There, and Where Shall We Go? , 2008, Industrial and Organizational Psychology.

[77]  Ronald F. Piccolo,et al.  The forgotten ones? The validity of consideration and initiating structure in leadership research. , 2004, The Journal of applied psychology.

[78]  P. Bentler,et al.  Cutoff criteria for fit indexes in covariance structure analysis : Conventional criteria versus new alternatives , 1999 .

[79]  Robert G. Jones,et al.  Assessor Constructs in Use as the Missing Component in Validation of Assessment Center Dimensions: A Critique and Directions for Research , 2008 .

[80]  Anton J. Villado,et al.  The importance of distinguishing between constructs and methods when comparing predictors in personnel selection research and practice. , 2008, The Journal of applied psychology.