Refining Situational Judgment Test Methods

Abstract : Situational Judgment Tests (SJTs) assess knowledge, skills, values, and attitudes. They present scenarios, which are based on real events, to be judged, understood, scaled, and interpreted by the examinee. These instruments have been used to evaluate cognitive theories and training programs, and to predict performance. This paper presents the argument that SJTs have potential to renew and reinvigorate many aspects of psychological measurement. We provide a framework to categorize the broad range of procedures and formats adopted for SJTs. The framework indicates that the psychometric range and power of SJTs might be extended by incorporating: (a) Theories and models of human cognition and performance to systematically specify the detail provided in the scenarios; (b) Likert and constructed response formats to maximize breadth of information collected for each scenario; and (c) Consensus-based scoring methods to evaluate knowledge and attitude domains associated with emerging applications.

[1]  S. Urbina,et al.  Psychological testing, 7th ed. , 1997 .

[2]  Lynn A. Streeter,et al.  Augmenting Electronic Environments for Leadership , 2004 .

[3]  J. Carroll.  Human Cognitive Abilities , 1993 .

[4]  Risto Näätänen,et al.  A model for the role of motivational factors in drivers' decision-making , 1974 .

[5]  John P. Campbell,et al.  Exploring the Limits in Personnel Selection and Classification , 2001 .

[6]  T. Mitchell,et al.  The Conditional Reasoning Measurement System for Aggression: An Overview , 2004, Human Performance.

[7]  Robert J. Townsley,et al.  What is a Good? , 1999 .

[8]  Leonard Evans,et al.  Traffic Safety and the Driver , 1991 .

[9]  Daniel E. Martin,et al.  Measuring Cognitive Aptitude Using Unobtrusive Knowledge Tests: A New Survey Technology , 2000 .

[10]  A. Jensen,et al.  What is a good g , 1994 .

[11]  Pamela J. Hinds,et al.  The curse of expertise: The effects of expertise and debiasing methods on prediction of novice performance. , 1999 .

[12]  Leonard Evans,et al.  Human behavior and traffic safety , 2011 .

[13]  R. Näätänen,et al.  Road user behavior and traffic accidents , 1976 .

[14]  N. Nguyen,et al.  Situational Judgment Tests: A Review of Practice and Constructs Assessed , 2001 .

[15]  Lawrence R. James,et al.  Measurement of Personality via Conditional Reasoning , 1998 .

[16]  M. D. Dunnette,et al.  An alternative selection procedure: The low-fidelity simulation. , 1990 .

[17]  Daniel E. Martin,et al.  Traffic crash involvement: experiential driving knowledge and stressful contextual antecedents. , 2003, The Journal of applied psychology.

[18]  M. Polanyi.  Chapter 7 – The Tacit Dimension , 1997 .

[19]  P. Legree.  Evidence for an oblique social intelligence factor established with a Likert-based testing procedure , 1995 .

[20]  M. Chi,et al.  The Nature of Expertise , 1988 .

[21]  Robert J. Sternberg,et al.  Practical intelligence in real-world pursuits: The role of tacit knowledge. , 1985 .

[22]  Robert M. Kaplan,et al.  Psychological testing: Principles, applications, and issues, 5th ed. , 2001 .

[23]  P. Salovey,et al.  Emotional intelligence meets traditional standards for an intelligence , 1999 .

[24]  Nancy Tippins,et al.  Further studies of the low‐fidelity simulation in the form of a situational inventory , 1993 .

[25]  P. Salovey,et al.  Measuring emotional intelligence with the MSCEIT V2.0. , 2003, Emotion.

[26]  Projective Methods for the Study of Personality , 1939 .

[27]  F. Hayek.  Individualism and Economic Order , 1949 .

[28]  Bias in Mental Testing , 1981 .

[29]  K. VanLehn.  Mind Bugs: The Origins of Procedural Misconceptions , 1990 .

[30]  Allen Newell,et al.  Human Problem Solving. , 1973 .

[31]  Michael A. Campion,et al.  Use of situational judgment tests to predict job performance: a clarification of the literature. , 2001, The Journal of applied psychology.

[32]  W. Arthur,et al.  Prediction of Vehicular Accident Involvement: A Meta-Analysis , 1991 .

[33]  Heikki Summala,et al.  Modeling Driver Behavior: A Pessimistic Prediction , 1985 .