The operational validity of a video-based situational judgment test for medical college admissions: illustrating the importance of matching predictor and criterion construct domains.

This study is part of a trend of examining noncognitive predictors, for example, a situational judgment test (SJT), as supplements to cognitive predictors for making college admission decisions. The authors examined criterion data over multiple academic years and universities. The criterion domain was broadly conceptualized, including both cognitive and interpersonal domains. The sample consisted of 7,197 candidates of the Medical and Dental Studies Admission Exam in Belgium. Results confirmed the importance of cognitive predictors. A video-based SJT was differentially valid for predicting overall grade point average for different curricula. The SJT showed incremental validity over cognitively oriented measures for curricula that included interpersonal courses, but not for other curricula. The SJT became more valid through the years. This demonstrates the importance of carefully specifying predictor- criterion linkages and of differentiating both predictor and criterion constructs.

[1]  Jeff W. Johnson,et al.  The relative importance of task and contextual performance dimensions to supervisor judgments of overall performance. , 2001, The Journal of applied psychology.

[2]  R. Roessler,et al.  Cognitive and noncognitive variables in the prediction of preclinical performance. , 1978, Journal of medical education.

[3]  Kevin R. Murphy,et al.  Implications of the multidimensional nature of job performance for the validity of selection tests , 1997 .

[4]  D. Powis.  Selecting medical students , 2008, The Medical journal of Australia.

[5]  C. Cooper,et al.  International review of industrial and organizational psychology , 1986 .

[6]  R. Reiter‐Palmon,et al.  Item selection counts: a comparison of empirical key and rational scale validities in theory-based and non-theory-based item pools. , 2000, The Journal of applied psychology.

[7]  P. Jansen,et al.  The dynamics of assessment center validity: results of a 7-year study. , 2001, The Journal of applied psychology.

[8]  E. F. Stone-Romero,et al.  Estimating the Power to Detect Dichotomous Moderators with Moderated Multiple Regression , 1994 .

[9]  Jill E. Ellingson,et al.  High-stakes testing in employment, credentialing, and higher education. Prospects in a post-affirmative-action world. , 2001, The American psychologist.

[10]  D. Webster,et al.  Preclinical progress in relation to personality and academic profiles , 1993, Medical education.

[11]  Keith Hattrup,et al.  The effects of varying conceptualizations of job performance on adverse impact, minority hiring, and predicted performance. , 1997 .

[12]  M. Ree,et al.  Sign changes when correcting for range restriction: A note on Pearson's and Lawley's selection formulas. , 1994 .

[13]  Jeffrey J. Mchenry,et al.  PROJECT A VALIDITY RESULTS: THE RELATIONSHIP BETWEEN PREDICTOR AND CRITERION DOMAINS , 1990 .

[14]  J. Fleiss,et al.  Intraclass correlations: uses in assessing rater reliability. , 1979, Psychological bulletin.

[15]  N. Schmitt,et al.  Personnel Selection: A Theoretical Approach , 1998 .

[16]  Sarah A. Hezlett,et al.  A comprehensive meta-analysis of the predictive validity of the graduate record examinations: implications for graduate student selection and performance. , 2001, Psychological bulletin.

[17]  N. Nguyen,et al.  Situational Judgment Tests: A Review of Practice and Constructs Assessed , 2001 .

[18]  Gillian King,et al.  Personality and Cognitive Ability Predictors of Performance in Graduate Business School. , 1994 .

[19]  Jeff A. Weekley,et al.  Video-based situational testing. , 1997 .

[20]  Michael A. McDaniel,et al.  Use of situational judgment tests to predict job performance: a clarification of the literature. , 2001, The Journal of applied psychology.

[21]  N. Schmitt,et al.  Video-based versus paper-and-pencil method of assessment in situational judgment tests: subgroup differences in test performance and face validity perceptions. , 1997, The Journal of applied psychology.

[22]  Nancy Tippins,et al.  Further studies of the low‐fidelity simulation in the form of a situational inventory , 1993 .

[23]  g: Is This Your Final Answer? , 2002 .

[24]  M. D. Dunnette,et al.  An alternative selection procedure: The low-fidelity simulation. , 1990 .

[25]  W. D. Corte.  Weighing job performance predictors to both maximize the quality of the selected workforce and control the level of adverse impact , 1999 .

[26]  M. Rotundo.  The relative importance of task, citizenship, and counterproductive performance to global ratings of job performance: a policy-capturing approach. , 2002, The Journal of applied psychology.

[27]  N. Schmitt,et al.  Situational Judgment and Job Performance , 2002 .

[28]  C. Teutsch.  Patient-doctor communication. , 2003, The Medical clinics of North America.

[29]  K. Mitchell,et al.  Assessing the validity of the updated Medical College Admission Test , 1994, Academic medicine : journal of the Association of American Medical Colleges.

[30]  Morell E. Mullins,et al.  PARALLEL TEST FORM DEVELOPMENT: A PROCEDURE FOR ALTERNATE PREDICTORS AND AN EXAMPLE , 1998 .

[31]  Joseph G. Rosse,et al.  The impact of response distortion on preemployment personality testing and hiring decisions. , 1998 .

[32]  W. Borman,et al.  Expanding the Criterion Domain to Include Elements of Contextual Performance , 1993 .

[33]  K. Murphy.  Personnel Selection in Organizations , 1992 .

[34]  N. Schmitt,et al.  Developing a biodata measure and situational judgment inventory as predictors of college student performance. , 2004, The Journal of applied psychology.

[35]  R. Tomlinson,et al.  The relative role of ‘A’ level chemistry, physics and biology in the medical course , 1977, Medical education.

[36]  R. Garnish.  A-level grades and medical school admission , 1982 .

[37]  P. Sackett,et al.  Correction for range restriction: an expanded typology. , 2000, The Journal of applied psychology.

[38]  Filip Lievens,et al.  Situational Tests in Student Selection: An Examination of Predictive Validity, Adverse Impact, and Construct Validity , 2002 .

[39]  F. Lievens,et al.  Situational Judgment Tests and their Predictiveness of College Students’ Success: The Influence of Faking , 2005 .

[40]  Jeff A. Weekley,et al.  FURTHER STUDIES OF SITUATIONAL TESTS , 1999 .

[41]  T J Peters,et al.  An assessment of academic performance and personality , 1991, Medical education.

[42]  W. Montague,et al.  Academic selection criteria and subsequent performance , 1990, Medical education.

[43]  Neal Schmitt,et al.  Incremental validity of situational judgment tests , 2001 .

[44]  Milton D. Hakel,et al.  Asking the Right Questions About g , 2002 .

[45]  P. Blumberg.  Multidimensional Outcome Considerations in Assessing the Efficacy of Medical Educational Programs , 2003, Teaching and learning in medicine.

[46]  Paul Jansen,et al.  The dynamics of assessment center validity: results of a seven year study , 2001 .