EVALUATING INSTRUCTOR/EVALUATOR INTER-RATER RELIABILITY FROM PERFORMANCE DATABASE INFORMATION

Carriers operating under the FAA's Advanced Qualification Program are required to assess individual and crew performance via Line Oriented Evaluations (LOEs). LOEs take place in a full-motion simulator, and involve a full crew performing a simulated flight from take-off to landing. Evaluating crew performance in the LOE is an arduous task, even for highly-trained professionals. Therefore, techniques are needed for training Instructor/Evaluators (I/Es), and for maintaining I/E calibration indefinitely. This paper describes the major steps involved in the development of Inter-Rater Reliability (IRR) training programs, as well as the usefulness of LOE performance database information for assessing I/E calibration between IRR training sessions. BACKGROUND