Does Situation Awareness Add to the Validity of Cognitive Tests?

Objective: Does adding situation awareness (SA) to a battery of cognitive tests improve prediction? Background: Identifying variables that predict skilled performance in a complex task aids in understanding the nature of skill and also aids in the selection of operators to perform that task. SA is thought to be an important predictor of performance. SA is often thought to be based on underlying cognitive mechanisms. Method: Three performance measures taken from the Federal Aviation Administration's (FAA's) Air Traffic Scenarios Test, the low-fidelity simulation component of the FAA's controller selection battery, were used as criterion variables in a hierarchical regression. After predicting performance based on a battery of cognitive (e.g., intelligence, working memory, spatial memory) and noncognitive tests (e.g., cognitive style, personality, demographics), we added measures of SA. Results: SA did provide increases in prediction, but only when measured with the Situation Present Assessment Method, an on-line query method. When the same questions were asked off line, SA did not enter the model in two cases and improved prediction by only 2% in the third. Conclusion: Thus, some measures of SA do show incremental validity, even against a backdrop of a large number of cognitive variables. Application: On-line measures of SA can be a worthwhile addition to standard batteries of tests used to predict performance in cognitively oriented industrial tasks.

[1]  E. Jeannot,et al.  The Development of Situation Awareness Measures in ATM Systems , 2003 .

[2]  P. Ackerman,et al.  Ability and task constraint determinants of complex task performance. , 2002, Journal of experimental psychology. Applied.

[3]  Diane F. Halpern,et al.  Sex Differences in Cognitive Abilities. Third Edition. , 2000 .

[4]  R. Cattell.  Theory of fluid and crystallized intelligence: A critical experiment. , 1963 .

[5]  Debra Jones,et al.  Situation Awareness Measurement and the Situation Awareness Global Assessment Technique , 2004 .

[6]  R. Engle,et al.  Is working memory capacity task dependent , 1989 .

[7]  S. Dumais,et al.  Handbook of applied cognition , 2007 .

[8]  D. Halpern.  Sex Differences in Cognitive Abilities , 1986 .

[9]  Mica R. Endsley,et al.  Toward a Theory of Situation Awareness in Dynamic Systems , 1995, Hum. Factors.

[10]  J. Carroll.  Human Cognitive Abilities-a sur-vey of factor-analytic studies , 1993 .

[11]  P. Ackerman.  Traits and knowledge as determinants of learning and individual differences: Putting it all together. , 1999 .

[12]  A. D. Fisk,et al.  Do ability-performance relationships differentiate age and practice effects in visual search? , 1994, Journal of experimental psychology. Learning, memory, and cognition.

[13]  Robert H. Logie,et al.  Visuo-spatial and verbal working memory in the five-disc Tower of London task: An individual differences approach , 2002 .

[14]  Dana Broach,et al.  The Effects of Video Game Experience on Computer-Based Air Traffic Controller Specialist, Air Traffic Scenario Test Scores. , 1997 .

[15]  Carol A. Manning,et al.  Documentation of validity for the AT-SAT computerized test battery, volume II.: (446082004-001) , 2001 .

[16]  R. Linn.  Educational measurement, 3rd ed. , 1989 .

[17]  V. David Hopkin.  Human Factors In Air Traffic Control , 1995 .

[18]  Stephanie M. Doane,et al.  Memory Processes of Flight Situation Awareness: Interactive Roles of Working Memory Capacity, Long-Term Working Memory, and Expertise , 2004, Hum. Factors.

[19]  Hinnerk Eißfeldt,et al.  Staffing the ATM System: The Selection of Air Traffic Controllers , 2002 .

[20]  S B Sells,et al.  Selection of Air Traffic Controllers , 1984 .

[21]  Susy Macqueen,et al.  Validity , 1973, Just Algorithms.

[22]  Kristan J. Wheaton.  The Nature of Intelligence , 2001 .

[23]  Michael C. Heil,et al.  The Effects of Practice and Coaching on the Air Traffic Selection and Training Test Battery , 2002 .

[24]  Karl G. Jöreskog,et al.  Lisrel 8: Structural Equation Modeling With the Simplis Command Language , 1993 .

[25]  Cheryl A. Bolstad,et al.  Individual Pilot Differences Related to Situation Awareness , 1991 .

[26]  Murray R. Barrick,et al.  THE BIG FIVE PERSONALITY DIMENSIONS AND JOB PERFORMANCE: A META-ANALYSIS , 1991 .

[27]  Mica R. Endsley,et al.  Experimental Analysis and Measurement of Situation Awareness , 1995 .

[28]  Randall W Engle,et al.  Working memory, short-term memory, and general fluid intelligence: a latent-variable approach. , 1999, Journal of experimental psychology. General.

[29]  J Cerella,et al.  Cognitive skill acquisition and transfer in younger and older adults. , 2001, Psychology and aging.

[30]  Diane F. Halpern,et al.  Sex differences in cognitive abilities, 2nd ed. , 1992 .

[31]  W. C. Shipley.  Shipley Institute of Living Scale , 1983 .

[32]  Mica R. Endsley,et al.  Measurement of Situation Awareness in Dynamic Systems , 1995, Hum. Factors.

[33]  P. Ackerman.  Determinants of individual differences during skill acquisition: Cognitive abilities and information processing. , 1988 .

[34]  R. Engle,et al.  Working-memory capacity, proactive interference, and divided attention: limits on long-term memory retrieval. , 2000, Journal of experimental psychology. Learning, memory, and cognition.

[35]  P. Carpenter,et al.  Individual differences in working memory and reading , 1980 .

[36]  S. Gronlund,et al.  Situation Awareness , 2006 .

[37]  H A WITKIN.  USES OF THE CENTRIFUGE IN STUDIES OF THE ORIENTATION OF SPACE. , 1964, The American journal of psychology.

[38]  Andrew J Tattersall,et al.  INDIVIDUAL DIFFERENCES IN PERFORMANCE. , 1998 .

[39]  Thomas R. Carretta,et al.  Pilot selection methods. , 2000 .

[40]  Department of Transportation Federal Aviation Administration 14 Cfr Part 91 Automatic Dependent Surveillance— Broadcast (ads–b) out Performance Requirements to Support Air Traffic Control (atc) Service; Final Rule Department of Transportation Federal Aviation Administration 14 Cfr Part 91 Automatic , 2022 .

[41]  L. R. Goldberg.  THE DEVELOPMENT OF MARKERS FOR THE BIG-FIVE FACTOR STRUCTURE , 1992 .

[42]  Michael C. Pyryt.  Human cognitive abilities: A survey of factor analytic studies , 1998 .

[43]  T R Carretta,et al.  Prediction of situational awareness in F-15 pilots. , 1996, The International journal of aviation psychology.

[44]  J. Raven,et al.  The Raven Progressive Matrices: A Review of National Norming Studies and Ethnic and Socioeconomic Variation Within the United States , 1989 .

[45]  J. W. French.  The description of aptitude and achievement tests in terms of rotated factors. , 1951 .

[46]  J. Cacioppo,et al.  The need for cognition. , 1982 .

[47]  A. D. Fisk,et al.  Aging and individual differences in the development of skilled memory search performance. , 1996, Psychology and aging.

[48]  S. Tremblay.  A Cognitive Approach to Situation Awareness: Theory and Application , 2004 .

[49]  A. Miyake,et al.  The separability of working memory resources for spatial thinking and language processing: an individual differences approach. , 1996, Journal of experimental psychology. General.

[50]  Danko Nikolić,et al.  Expertise and chess: A pilot study comparing situation awareness methodologies , 1995 .

[51]  Richard D. Roberts,et al.  Learning and Individual Differences: Process, Trait and Content Determinants , 1999 .

[52]  Francis T. Durso,et al.  En Route Operational Errors and Situational Awareness , 1998 .

[53]  M. D’Esposito.  Working memory. , 2008, Handbook of clinical neurology.

[54]  R. Engle,et al.  The generality of working memory capacity: a latent-variable approach to verbal and visuospatial memory span and reasoning. , 2004, Journal of experimental psychology. General.

[55]  Raymond B. Cattell,et al.  Models and Paradigms in Personality and Intelligence Research , 1995 .

[56]  P. Costa,et al.  Domains and facets: hierarchical personality assessment using the revised NEO personality inventory. , 1995, Journal of personality assessment.