Towards Multi-UAV and Human Interaction Driving System Exploiting Human Mental State Estimation

This paper addresses the growing human-multi-UAV interaction issue. Current active approaches towards a reliable multi-UAV system are reviewed. This brings us to the conclusion that the multiple Unmanned Aerial Vehicles (UAVs) control paradigm is segmented into two main scopes: i) autonomous control and coordination within the group of UAVs, and ii) a human centered approach with helping agents and overt behavior monitoring. Therefore, to move further with the future of human-multi-UAV interaction problem, a new perspective is put forth. In the following sections, a brief understanding of the system is provided, followed by the current state of multi-UAV research and how taking the human pilot's physiology into account could improve the interaction. This idea is developed first by detailing what physiological computing is, including mental states of interest and their associated physiological markers. Second, the article concludes with the proposed approach for Human-multi-UAV interaction control and future plans.

[1]  Michelle N. Lumicao,et al.  EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks. , 2007, Aviation, space, and environmental medicine.

[2]  Frédéric Dehais,et al.  Mixed-initiative mission planning considering human operator state estimation based on physiological sensors , 2017 .

[3]  Stephen H. Fairclough,et al.  Evaluation of an Adaptive Game that Uses EEG Measures Validated during the Design Process as Inputs to a Biocybernetic Loop , 2016, Front. Hum. Neurosci..

[4]  Antoine Drouin,et al.  The Paparazzi Solution , 2006 .

[5]  Ronald C. Arkin,et al.  Mixed-Initiative Human-Robot Interaction: Definition, Taxonomy, and Survey , 2015, 2015 IEEE International Conference on Systems, Man, and Cybernetics.

[6]  Aníbal Ollero,et al.  Journal of Intelligent & Robotic Systems manuscript No. (will be inserted by the editor) Experimental Results in Multi-UAV Coordination for Disaster Management and Civil Security Applications , 2022 .

[7]  I. Parberry,et al.  Evaluating player task engagement and arousal using electroencephalography , 2015 .

[8]  R. Parasuraman,et al.  Psychophysiology and adaptive automation , 1996, Biological Psychology.

[9]  Danielle S. McNamara,et al.  Interference Effects on the Recall of Words Heard and Read: Considerations for ATC Communication , 2002 .

[10]  Shayne Loft,et al.  Prospective memory and task interference in a continuous monitoring dynamic display task. , 2010, Journal of experimental psychology. Applied.

[11]  Stéphane Bonnet,et al.  Efficient Workload Classification based on Ignored Auditory Probes: A Proof of Concept , 2016, Front. Hum. Neurosci..

[12]  Paolo Bonato,et al.  Neuroadaptive technologies: Applying neuroergonomics to the design of advanced interfaces , 2003 .

[13]  Ugur Kuter,et al.  A Human-System Interface with Contingency Planning for Collaborative Operations of Unmanned Aerial Vehicles , 2017 .

[14]  Loukia D. Loukopoulos,et al.  COCKPIT INTERRUPTIONS AND DISTRACTIONS: A LINE OBSERVATION STUDY , 2001 .

[15]  Julie A. Adams,et al.  A Survey of Workload Assessment Algorithms , 2018, IEEE Transactions on Human-Machine Systems.

[16]  G. Vachtsevanos,et al.  Future of Unmanned Aviation , 2014 .

[17]  Ranjana K. Mehta,et al.  Neuroergonomics: a review of applications to physical and cognitive work , 2013, Front. Hum. Neurosci..

[18]  Thibault Gateau,et al.  Operator Engagement During Prolonged Simulated UAV Operation , 2016 .

[19]  Aníbal Ollero,et al.  A Ground Control Station for a Multi-UAV Surveillance System , 2013, J. Intell. Robotic Syst..

[20]  Raja Parasuraman,et al.  Event-Related Potentials (ERPs) in Neuroergonomics , 2006, Neuroergonomics.

[21]  Antonio Franchi,et al.  Shared Control : Balancing Autonomy and Human Assistance with a Group of Quadrotor UAVs , 2012, IEEE Robotics & Automation Magazine.

[22]  Frédéric Dehais,et al.  MOMDP-Based Target Search Mission Taking into Account the Human Operator's Cognitive State , 2015, 2015 IEEE 27th International Conference on Tools with Artificial Intelligence (ICTAI).

[23]  Axel Schulte,et al.  Single-Seat Cockpit-Based Management of Multiple UCAVs Using On-Board Cognitive Agents for Coordination in Manned-Unmanned Fighter Missions , 2013, HCI.

[24]  Claude Frasson,et al.  Mental Workload, Engagement and Emotions: An Exploratory Study for Intelligent Tutoring Systems , 2012, ITS.

[25]  Thibault Gateau,et al.  Considering human's non-deterministic behavior and his availability state when designing a collaborative human-robots system , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[26]  Bernhard Rinner,et al.  An Autonomous Multi-UAV System for Search and Rescue , 2015, DroNet@MobiSys.

[27]  Kevin W Williams,et al.  A Summary of Unmanned Aircraft Accident/Incident Data: Human Factors Implications , 2004 .

[28]  Julie A. Adams,et al.  A Diagnostic Human Workload Assessment Algorithm for Human-Robot Teams , 2018, HRI.

[29]  Christopher Bolkcom,et al.  Homeland Security: Unmanned Aerial Vehicles and Border Surveillance , 2004 .

[30]  Axel Schulte,et al.  Management of multiple unmanned combat aerial vehicles from a single-seat fighter cockpit in manned-unmanned fighter missions , 2013 .

[31]  Robin R. Murphy,et al.  Human-robot interactions during the robot-assisted urban search and rescue response at the World Trade Center , 2003, IEEE Trans. Syst. Man Cybern. Part B.

[32]  Glenn F. Wilson,et al.  Workload assessment in multi-task environments , 2020 .

[33]  Stephen H. Fairclough,et al.  Fundamentals of physiological computing , 2009, Interact. Comput..

[34]  Axel Schulte,et al.  Human-System Interaction Analysis for Military Pilot Activity and Mental Workload Determination , 2015, 2015 IEEE International Conference on Systems, Man, and Cybernetics.

[35]  A. Pope,et al.  Biocybernetic system evaluates indices of operator engagement in automated task , 1995, Biological Psychology.

[36]  Gautier Hattenberger,et al.  Multi-UAV control with the Paparazzi system , 2008 .

[37]  Sergi Bermúdez i Badia,et al.  Physiologically Attentive User Interface for Robot Teleoperation - Real Time Emotional State Estimation and Interface Modification using Physiology, Facial Expressions and Eye Movements , 2019, BIOSIGNALS.

[38]  Sidney Givigi,et al.  Towards human-robot interaction: A framing effect experiment , 2016, 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[39]  S Bonnet,et al.  Efficient mental workload estimation using task-independent EEG features , 2016, Journal of neural engineering.

[40]  Milind Tambe,et al.  Improving adjustable autonomy strategies for time-critical domains , 2009, AAMAS.

[41]  Thibault Gateau,et al.  In silico vs. Over the Clouds: On-the-Fly Mental State Estimation of Aircraft Pilots, Using a Functional Near Infrared Spectroscopy Based Passive-BCI , 2018, Front. Hum. Neurosci..

[42]  Frédéric Dehais,et al.  Pre-stimulus antero-posterior EEG connectivity predicts performance in a UAV monitoring task , 2017, 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[43]  Daniel Gopher,et al.  Workload: An examination of the concept. , 1986 .