Development and Validation of the Controller Acceptance Rating Scale (CARS): Results of Empirical Research

The measurement of operational acceptability is important for the development, implementation, and evolution of air traffic management decision support tools. The Controller Acceptance Rating Scale was developed at NASA Ames Research Center for the development and evaluation of the Passive Final Approach Spacing Tool. CARS was modeled after a well-known pilot evaluation rating instrument, the Cooper-Harper Scale, and has since been used in the evaluation of the User Request Evaluation Tool, developed by MITRE's Center for Advanced Aviation System Development. In this paper, we provide a discussion of the development of CARS and an analysis of the empirical data collected with CARS to examine construct validity. Results of intraclass correlations indicated statistically significant reliability for the CARS. From the subjective workload data that were collected in conjunction with the CARS, it appears that the expected set of workload attributes was correlated with the CARS. As expected, the analysis also showed that CARS was a sensitive indicator of the impact of decision support tools on controller operations. Suggestions for future CARS development and its improvement are also provided.

[1]  George E. Cooper,et al.  Handling qualities and pilot evaluation , 1986 .

[2]  Thomas L. Seamster,et al.  Evaluation of Aircraft Pilot Team Performance , 1996 .

[3]  Raja Parasuraman,et al.  Humans and Automation: Use, Misuse, Disuse, Abuse , 1997, Hum. Factors.

[4]  Fred Schick.  Methods and Measurement for the Evaluation of ATM Tools in Real-Time Simulations and Field Tests , 1998 .

[5]  J. Fleiss,et al.  Intraclass correlations: uses in assessing rater reliability. , 1979, Psychological bulletin.

[6]  Karol Kerns.  An experimental approach to measuring the effects of a controller conflict probe in a free routing environment , 2001, IEEE Trans. Intell. Transp. Syst..

[7]  David J. Wilson,et al.  Cooper-Harper pilot rating variability , 1989 .

[8]  John E. Deaton,et al.  AVIATION RESEARCH AND DEVELOPMENT: A FRAMEWORK FOR THE EFFECTIVE PRACTICE OF HUMAN FACTORS, OR "WHAT YOUR MENTOR NEVER TOLD YOU ABOUT A CAREER IN HUMAN FACTORS...". IN: HANDBOOK OF AVIATION HUMAN FACTORS , 1999 .

[9]  L. James,et al.  rwg: An assessment of within-group interrater agreement. , 1993 .

[10]  David G. Mitchell,et al.  Reassessment and extensions of pilot ratings with new data , 1990 .

[11]  John E. Robinson,et al.  Operational Test Results of the Final Approach Spacing Tool , 1996 .

[12]  D. Campbell,et al.  Convergent and discriminant validation by the multitrait-multimethod matrix. , 1959, Psychological bulletin.

[13]  K. McGraw,et al.  Forming inferences about some intraclass correlation coefficients. , 1996 .

[14]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[15]  Jeffery A. Schroeder,et al.  Simulator Platform Motion Effects on Pilot-Induced Oscillation Prediction , 2000 .

[16]  Katharine K. Lee,et al.  Human Factors Assessment: The Passive Final Approach Spacing Tool (pFAST) Operational Evaluation , 1998 .

[17]  John G. Casali,et al.  A Validated Rating Scale for Global Mental Workload Measurement Applications , 1983 .

[18]  K. Hendy,et al.  Measuring Subjective Workload: When Is One Scale Better Than Many? , 1993 .

[19]  George E. Cooper,et al.  The use of pilot rating in the evaluation of aircraft handling qualities , 1969 .

[20]  KATHARINE,et al.  The Development of the Final Approach Spacing Tool (FAST): A Cooperative Controller- Engineer Design Approach , 1995 .

[21]  Alvin L. McFarland,et al.  Preliminary Observations About Providing Problem Resolution Advisories to Air Traffic Controllers , 2000 .

[22]  David L. Green,et al.  Interpreted Cooper-Harper for broader use , 1993 .

[23]  Katharine K. Lee,et al.  The development of the Final Approach Spacing Tool (FAST): a cooperative controller-engineer design approach , 1995 .

[24]  Paul J. Sherman,et al.  DO RATERS AGREE? -- ASSESSING INTER-RATER AGREEMENT IN THE EVALUATION AND AIR CREW RESOURCE MANAGEMENT SKILLS. , 1995 .