Using Item Response Theory to Evaluate Measurement Precision of Selection Tests at the French Pilot Training

In pilot selection settings, decisions are often based on cutoff scores. In item response theory the measurement precision of a test score can be evaluated by its degree of information. We investigated whether the maximum of test information corresponded to the cutoff zone for 10 cognitive ability tests of the current French civil air transport pilot selection (n = 577). An item response theory model was fitted to the data. The 6 adequately fitted tests' test information curves were generally maximal in the cutoff score zone. Several ways to improve the absolute level of precision of tests in pilot selection situations are discussed.

[1]  N. Cliff,et al.  Ordinal measurement in the behavioral sciences , 2002 .

[2]  D L Damos,et al.  Pilot selection batteries: shortcomings and perspectives. , 1996, The International journal of aviation psychology.

[3]  R. Hambleton,et al.  Fundamentals of Item Response Theory , 1991 .

[4]  S. Embretson,et al.  Item response theory for psychologists , 2000 .

[5]  F. Baker.  The basics of item response theory , 1985 .

[6]  N. D. Verhelst,et al.  A Logistic Model for Time-Limit Tests , 1992 .

[7]  M. Edelen,et al.  Applying item response theory (IRT) modeling to questionnaire development, evaluation, and refinement , 2007, Quality of Life Research.

[8]  Hinnerk Eißfeldt,et al.  Ability requirements in core aviation profession. Job analyses of airline pilots and air traffic controllers. , 2004 .

[9]  M Martinussen,et al.  Pilot selection in the Norwegian Air Force: a validation and meta-analysis of the test battery. , 1997, The International journal of aviation psychology.

[10]  E Burke,et al.  Large sample validations of three general predictors of pilot training success. , 1997, The International journal of aviation psychology.

[11]  Georg Rasch,et al.  Probabilistic Models for Some Intelligence and Attainment Tests , 1981, The SAGE Encyclopedia of Research Design.

[12]  R. Key Dismukes,et al.  Pilot Instructor Rater Training: The Utility of the Multifacet Item Response Theory Model , 2002 .

[13]  Melvin R. Novick,et al.  Some latent train models and their use in inferring an examinee's ability , 1966 .

[14]  S. Vautier,et al.  Test-Specificity of the Advantage of Retaking Cognitive Ability Tests , 2011 .

[15]  Markus Sommer,et al.  Improvements in Personnel Selection With Neural Networks: A Pilot Study in the Field of Aviation Psychology , 2004 .

[16]  E. Raufaste,et al.  Situational effects may account for gain scores in cognitive ability testing: A longitudinal SEM approach , 2009 .

[17]  Fritz Drasgow,et al.  Recovery of Two- and Three-Parameter Logistic Item Characteristic Curves: A Monte Carlo Study , 1982 .

[18]  Fritz Drasgow,et al.  Fitting Polytomous Item Response Theory Models to Multiple-Choice Tests , 1995 .