Cognitive Human-Machine Interfaces and Interactions for Unmanned Aircraft

This paper presents the concept of Cognitive Human-Machine Interfaces and Interactions (CHMI2) for Unmanned Aircraft System (UAS) Ground Control Stations (GCS). CHMI2 represents a new approach to aviation human factors engineering that introduces adaptive functionalities in the design of operators’ command, control and display functions. A CHMI2 system assesses human cognitive states based on measurement of key psycho-physiological observables. The cognitive states are used to predict and enhance operator performance in the accomplishment of aviation tasks, with the objective of improving the efficiency and effectiveness of the overall human-machine teaming. The CHMI2 system presented in this paper employs a four-layer architecture comprising sensing, extraction, classification and adaptation functionalities. An overview of each layer is provided along with the layer’s metrics, algorithms and functions. Two relevant case studies are presented to illustrate the interactions between the different layers, and the conceptual design of the associated display formats is described. The results indicate that specific eye tracking variables provide discrimination between different modes of control. Furthermore, results indicate that the higher levels of automation supported by the CHMI2 are beneficial in Separation Assurance and Collision Avoidance (SA&CA) scenarios involving low-detectability obstacles and stringent time constraints to implement recovery manoeuvres. These preliminary results highlight that the introduction of CHMI2 functionalities in future UAS can significantly reduce reaction time and enhance operational effectiveness of unmanned aircraft response to collision and loss of separation events, as well as improve the overall safety and efficiency of operations.

[1]  Kimon P. Valavanis,et al.  On Integrating Unmanned Aircraft Systems into the National Airspace System: Issues, Challenges, Operational Restrictions, Certification, and Recommendations , 2011 .

[2]  Jason J. Ford,et al.  Airborne vision‐based collision‐detection system , 2011, J. Field Robotics.

[3]  Alessandro Gardi,et al.  UAS in the terminal area: Challenges and opportunities , 2016 .

[4]  Mackenzie G Glaholt,et al.  Eye Tracking in the Cockpit: a Review of the Relationships between Eye Movements and the Aviators Cognitive State , 2014 .

[5]  Keith S. Karn,et al.  Commentary on Section 4. Eye tracking in human-computer interaction and usability research: Ready to deliver the promises. , 2003 .

[6]  Jing Liu,et al.  Cognitive pilot-aircraft interface for single-pilot operations , 2016, Knowl. Based Syst..

[7]  Max Mulder,et al.  An Information-Center ed Analysis of the Tunnel-in-the-Sky Display, Part 3: The Interception of Curved Trajectories , 2006 .

[8]  Claude Frasson,et al.  Modeling mental workload using EEG features for intelligent systems , 2011, UMAP'11.

[9]  Tanja Schultz,et al.  Determine Task Demand from Brain Activity , 2008, BIOSIGNALS.

[10]  Andreas Henelius,et al.  Mental workload classification using heart rate metrics , 2009, 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[11]  A. A. Spady,et al.  Analytical techniques of pilot scanning behavior and their application , 1986 .

[12]  Robert J. K. Jacob,et al.  Eye tracking in human-computer interaction and usability research : Ready to deliver the promises , 2002 .

[13]  Joseph H. Goldberg,et al.  Identifying fixations and saccades in eye-tracking protocols , 2000, ETRA.

[14]  Qiang Ji,et al.  Cross-subject workload classification with a hierarchical Bayes model , 2012, NeuroImage.

[15]  Roberto Sabatini,et al.  A Novel GNSS Integrity Augmentation System for Civil and Military Aircraft , 2013 .

[16]  Jianhua Zhang,et al.  Neurophysiological Feature-Based Detection of Mental Workload by Ensemble Support Vector Machines , 2016 .

[17]  A. H. Roscoe.  Assessing pilot workload. Why measure heart rate, HRV and respiration? , 1992, Biological Psychology.

[18]  Alessandro Gardi,et al.  Cooperative and non-cooperative sense-and-avoid in the CNS+A context: A unified methodology , 2016, 2016 International Conference on Unmanned Aircraft Systems (ICUAS).

[19]  Susana Rubio,et al.  Evaluation of Subjective Mental Workload: A Comparison of SWAT, NASA‐TLX, and Workload Profile Methods , 2004 .

[20]  Jess Gilland.  Driving, eye -tracking and visual entropy: Exploration of age and task effects , 2008 .

[21]  Michela Terenzi,et al.  A Random Glance at the Flight Deck: Pilots' Scanning Strategies and the Real-Time Assessment of Mental Workload , 2007 .

[22]  Kelly Neville,et al.  A Cognitive Work Analysis to Identify Human-Machine Interface Design Challenges Unique to Uninhabited Aircraft Systems , 2012 .

[23]  Kenneth W. Bauer,et al.  Improving pilot mental workload classification through feature exploitation and combination: a feasibility study , 2005, Comput. Oper. Res..

[24]  Jason J. Ford,et al.  Field-of-view, detection range, and false alarm trade-offs in vision-based aircraft detection , 2012 .

[25]  Robert Du,et al.  Towards measurement of brain function in operational environments , 1995, Biological Psychology.

[26]  Nigel H. Lovell,et al.  Estimating cognitive workload using wavelet entropy-based features during an arithmetic task , 2013, Comput. Biol. Medicine.

[27]  Ronald H. Stevens,et al.  Integrating EEG Models of Cognitive Load with Machine Learning Models of Scientific Problem Solving , 2013 .

[28]  Yufeng Ke,et al.  Towards an effective cross-task mental workload recognition model using electroencephalography based on feature selection and support vector machine regression. , 2015, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[29]  Kimon P. Valavanis,et al.  Thoughts and Recommendations on a UAS Integration Roadmap , 2012 .

[30]  John Collura,et al.  Unmanned Aircraft System traffic management: Concept of operation and system architecture , 2016 .

[31]  José del R. Millán,et al.  Improving Human Performance in a Real Operating Environment through Real-Time Mental Workload Detection , 2007 .

[32]  Michael E. Smith,et al.  Neurophysiological measures of cognitive workload during human-computer interaction , 2003 .

[33]  Glenn F. Wilson,et al.  Real-Time Assessment of Mental Workload Using Psychophysiological Measures and Artificial Neural Networks , 2003, Hum. Factors.

[34]  S. P. Marshall,et al.  The Index of Cognitive Activity: measuring cognitive workload , 2002, Proceedings of the IEEE 7th Conference on Human Factors and Power Plants.

[35]  Zhiwei Zhu,et al.  Non-Intrusive Measurement of Workload in Real-Time , 2005 .

[36]  Christophe Bourdin,et al.  Effectiveness of Physiological and Psychological Features to Estimate Helicopter Pilots' Workload: A Bayesian Network Approach , 2013, IEEE Transactions on Intelligent Transportation Systems.

[37]  Jing Liu,et al.  LIDAR obstacle warning and avoidance system for unmanned aerial vehicle sense-and-avoid , 2016 .