Gaze-based interactions in the cockpit of the future: a survey

Flying an aircraft is a mentally demanding task where pilots must process a vast amount of visual, auditory and vestibular information. They have to control the aircraft by pulling, pushing and turning different knobs and levers, while knowing that mistakes in doing so can have fatal outcomes. Therefore, attempts to improve and optimize these interactions should not increase pilots’ mental workload. By utilizing pilots’ visual attention, gaze-based interactions provide an unobtrusive solution to this. This research is the first to actively involve pilots in the exploration of gaze-based interactions in the cockpit. By distributing a survey among 20 active commercial aviation pilots working for an internationally operating airline, the paper investigates pilots’ perception and needs concerning gaze-based interactions. The results build the foundation for future research, because they not only reflect pilots’ attitudes towards this novel technology, but also provide an overview of situations in which pilots need gaze-based interactions.

[1]  M. Endsley.  Situation Awareness In Aviation Systems , 1999 .

[2]  Christopher D. Wickens,et al.  Cognitive factors in aviation display design , 1998, 17th DASC. AIAA/IEEE/SAE. Digital Avionics Systems Conference. Proceedings (Cat. No.98CH36267).

[3]  Barbara G. Kanki,et al.  Crew resource management , 2010 .

[4]  Anthony D. Andre,et al.  Situation Awareness in an Augmented Reality Cockpit: Design, Viewpoints and Cognitive Glue , 2005 .

[5]  D. Newton,et al.  Current Status of Gaze Control Research and Technology Literature Review , 2017 .

[6]  John M. Flach,et al.  Information Processing in Aviation , 2010 .

[7]  Andrew T. Duchowski,et al.  PETMEI 2011: the 1st international workshop on pervasive eye tracking and mobile eye-based interaction , 2011, UbiComp '11.

[8]  Oleg Spakov,et al.  Fast gaze typing with an adjustable dwell time , 2009, CHI.

[9]  Yongjin Kwon,et al.  Improved Flight Technical Performance in Flight Decks Equipped With Synthetic Vision Information System Displays , 2004 .

[10]  Jean-Paul Imbert,et al.  Design Requirements to Integrate Eye Trackers in Simulation Environments: Aeronautical Use Case , 2015, KES-IDT.

[11]  Kyle K. E. Ellis,et al.  Eye tracking metrics for workload estimation in flight deck operations , 2009 .

[12]  Thibault Gateau,et al.  "Automation Surprise" in Aviation: Real-Time Solutions , 2015, CHI.

[13]  Roope Raisamo,et al.  Gaze gestures and haptic feedback in mobile devices , 2014, CHI.

[14]  Sandra G. Hart,et al.  Nasa-Task Load Index (NASA-TLX); 20 Years Later , 2006 .

[15]  Patrick Langdon,et al.  State-of-the-Art and Future Concepts for Interaction in Aircraft Cockpits , 2015, HCI.

[16]  Martin Raubal,et al.  Visualizing pilot eye movements for flight instructors , 2018, ETVIS@ETRA.

[17]  Kai Kunze,et al.  Reading interventions: tracking reading state and designing interventions , 2016, UbiComp Adjunct.

[18]  Mickaël Causse,et al.  Evaluation of head-free eye tracking as an input device for air traffic control , 2013, Ergonomics.

[19]  Sohel Merchant,et al.  Eye Movement Research in Aviation and Commercially Available Eye Trackers Today , 2001 .

[20]  Roman Bednarik,et al.  What do you want to do next: a novel approach for intent prediction in gaze-based interaction , 2012, ETRA.

[21]  C. W. Johnson.  Looking Beyond the Cockpit: Human Computer Interaction in the Causal Complexes of Aviation Accidents , 2004 .

[22]  Andrew T. Duchowski,et al.  Eye Tracking Methodology: Theory and Practice , 2003, Springer London.

[23]  Randall E. Bailey,et al.  A review of head-worn display research at NASA Langley Research Center , 2015, Defense + Security Symposium.

[24]  Gal Ziv,et al.  Gaze Behavior and Visual Attention: A Review of Eye Tracking Studies in Aviation , 2016 .

[25]  Leopold Petrus Vermeulen,et al.  Flying glass : a qualitative analysis of pilot perceptions of automated flight-decks after 20 years , 2009 .

[26]  Konrad Schindler,et al.  Gaze-Informed location-based services , 2017, Int. J. Geogr. Inf. Sci..

[27]  Martin Raubal,et al.  GeoGazemarks: providing gaze history for the orientation on small display maps , 2012, ICMI '12.

[28]  Mickaël Causse,et al.  Embedded eye tracker in a real aircraft: new perspectives on pilot/aircraft interaction monitoring , 2008 .

[29]  C. Wickens,et al.  Visual scanning and pilot expertise: the role of attentional flexibility and mental model development. , 1997, Aviation, space, and environmental medicine.

[30]  Vsevolod Peysakhovich,et al.  Pilot Flying and Pilot Monitoring’s Aircraft State Awareness During Go-Around Execution in Aviation: A Behavioral and Eye Tracking Study , 2017 .

[31]  Mickaël Causse,et al.  The Neuroergonomics of Aircraft Cockpits: The Four Stages of Eye-Tracking Integration to Enhance Flight Safety , 2018 .

[32]  John Paulin Hansen,et al.  Wrist-worn pervasive gaze interaction , 2016, ETRA.

[33]  Yanxia Zhang,et al.  SideWays: a gaze interface for spontaneous interaction with situated displays , 2013, CHI.

[34]  Florian Jentsch,et al.  Chapter 14 – Aviation Displays , 2010 .

[35]  Sohel Merchant,et al.  Applying eye tracking as an alternative approach for activation of controls and functions in aircraft , 2000, 19th DASC. 19th Digital Avionics Systems Conference. Proceedings (Cat. No.00CH37126).

[36]  Hsiu-Fang Hsieh,et al.  Three Approaches to Qualitative Content Analysis , 2005, Qualitative health research.

[37]  Fred H. Previc,et al.  The Effects of Sleep Deprivation on Flight Performance, Instrument Scanning, and Physiological Arousal in Pilots , 2009 .

[38]  Robert J. K. Jacob,et al.  What you look at is what you get: eye movement-based interaction techniques , 1990, CHI '90.

[39]  Daniel G. Morrow,et al.  Expertise Differences in Attentional Strategies Related to Pilot Decision Making , 2008, Hum. Factors.

[40]  Kristen Liggett.  Controls, Displays, and Crew Station Design , 2009 .

[41]  Mickaël Causse,et al.  GazeForm: Dynamic Gaze-adaptive Touch Surface for Eyes-free Interaction in Airliner Cockpits , 2018, Conference on Designing Interactive Systems.

[42]  Hans-Werner Gellersen,et al.  Pursuits: spontaneous interaction with displays based on smooth pursuit eye movement and moving targets , 2013, UbiComp.

[43]  Yves Gourinat,et al.  The role of Pilots' monitoring strategies in flight performance , 2016 .

[44]  Christopher D. Wickens,et al.  Pilots' Monitoring Strategies and Performance on Automated Flight Decks: An Empirical Study Combining Behavioral and Eye-Tracking Data , 2007, Hum. Factors.