The State-of-the-Art of Human–Drone Interaction: A Survey

Drones have expanded from military operations to performing a broad range of civilian applications. As drone usage increases, humans will interact with such systems more often, therefore, it is important to achieve a natural human-drone interaction. Although some knowledge can be derived from the field of human-robot interaction, drones can fly in a 3D space, which essentially changes how humans can interact with them, making human-drone interaction a field of its own. This paper is the first survey on the emerging field of human-drone interaction focusing on multi-rotor systems, providing an overview of existing literature and the current state of the art in the field. This work begins with an analysis and comparison of the drone models that are commonly used by end-users and researchers in the field of human-drone interaction. Following, the current state of the field is discussed, including the roles of humans in HDI, innovative control methods, remaining aspects of interaction, and novelty drone prototypes and applications. This paper concludes by presenting a discussion of current challenges and future work in the field of human-drone interaction.

[1]  J. Rekimoto,et al.  Flying head: a head motion synchronization mechanism for unmanned aerial vehicle control , 2013, CHI Extended Abstracts.

[2]  Kouta Minamizawa,et al.  Telexistence drone: design of a flight telexistence system for immersive aerial sports experience , 2015, AH.

[3]  Sarah Garcia,et al.  Brain eRacing: An Exploratory Study on Virtual Brain-Controlled Drones , 2019, HCI.

[4]  Morten Fjeld,et al.  A Drone Agent to Support a Clean Environment , 2015, HAI.

[5]  Keita Higuchi,et al.  HoverBall: augmented sports with a flying ball , 2014, AH.

[6]  Florian Mueller,et al.  Understanding the design of a flying jogging companion , 2014, UIST'14 Adjunct.

[7]  Adrian Stoica,et al.  Remote Control of Quadrotor Teams, Using Hand Gestures , 2014, 2014 9th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[8]  Hiroki Nozaki,et al.  Flying display: a movable display pairing projector and screen in the air , 2014, CHI Extended Abstracts.

[9]  James A. Landay,et al.  Drone Near Me , 2017, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[10]  Hassan Noura,et al.  A gesture based kinect for quadrotor control , 2015, 2015 International Conference on Information and Communication Technology Research (ICTRC).

[11]  Morten Fjeld,et al.  Exploring Proxemics for Human-Drone Interaction , 2017, HAI.

[12]  Ehud Sharlin,et al.  Collocated interaction with flying robots , 2011, 2011 RO-MAN.

[13]  Bonifaz Kaufmann,et al.  Natural Interaction Techniques for an Unmanned Aerial Vehicle System , 2017, IEEE Pervasive Computing.

[14]  Takashi Suehiro,et al.  Action elements of emotional body expressions for flying robots , 2016, 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[15]  Robin R. Murphy,et al.  Comfortable approach distance with small Unmanned Aerial Vehicles , 2013, 2013 IEEE RO-MAN.

[16]  Thorsten O. Zander,et al.  A Survey on Unmanned Aerial Vehicle Remote Control Using Brain–Computer Interface , 2018, IEEE Transactions on Human-Machine Systems.

[17]  Martin Molina,et al.  Natural user interfaces for human-drone multi-modal interaction , 2016, 2016 International Conference on Unmanned Aircraft Systems (ICUAS).

[18]  Bertrand Rivet,et al.  Drone, your brain, ring course: accept the challenge and prevail! , 2014, UbiComp Adjunct.

[19]  Albrecht Schmidt,et al.  Midair Displays: Concept and First Experiences with Free-Floating Pervasive Displays , 2014, PerDis.

[20]  Keita Higuchi,et al.  Flying sports assistant: external visual imagery representation for sports training , 2011, AH '11.

[21]  Tianwei Shi,et al.  Brain Computer Interface system based on indoor semi-autonomous navigation and motor imagery for Unmanned Aerial Vehicle control , 2015, Expert Syst. Appl..

[22]  Tatsuo Nakajima,et al.  KinecDrone: enhancing somatic sensation to fly in the sky with Kinect and AR.Drone , 2014, AH.

[23]  Markus Funk,et al.  Tactile Drones - Providing Immersive Tactile Feedback in Virtual Reality through Quadcopters , 2017, CHI Extended Abstracts.

[24]  Patrick Doherty,et al.  A UAV Search and Rescue Scenario with Human Body Detection and Geolocalization , 2007, Australian Conference on Artificial Intelligence.

[25]  Michael A. Goodrich,et al.  Human-Robot Interaction: A Survey , 2008, Found. Trends Hum. Comput. Interact..

[26]  Markus Funk Human-drone interaction , 2018, Interactions.

[27]  Jean-Luc Dugelay,et al.  Autonomous drones for assisting rescue services within the context of natural disasters , 2014, 2014 XXXIth URSI General Assembly and Scientific Symposium (URSI GASS).

[28]  Jzau-Sheng Lin,et al.  Implementing Remote Presence Using Quadcopter Control by a Non-Invasive BCI Device , 2015 .

[29]  Walid Saad,et al.  Beyond 5G With UAVs: Foundations of a 3D Wireless Cellular Network , 2018, IEEE Transactions on Wireless Communications.

[30]  Keita Higuchi,et al.  Flying head: A head-synchronization mechanism for flying telepresence , 2013, 2013 23rd International Conference on Artificial Reality and Telexistence (ICAT).

[31]  Markus Funk,et al.  DroneLandArt: landscape as organic pervasive display , 2016, PerDis.

[32]  Cesar Lucho,et al.  Daedalus: A sUAV for Human-Robot Interaction , 2014, 2014 9th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[33]  Florian Mueller,et al.  Joggobot: a flying robot as jogging companion , 2012, CHI Extended Abstracts.

[34]  Bertrand Rivet,et al.  Bidirectional feedback in motor imagery BCIs: learn to control a drone within 5 minutes , 2014, CHI Extended Abstracts.

[35]  Kiyoshi Kiyokawa,et al.  A Non-grounded and Encountered-type Haptic Display Using a Drone , 2016, SUI.

[36]  Ting Sun,et al.  Gesture-based piloting of an aerial robot using monocular vision , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[37]  Albrecht Schmidt,et al.  Midair displays: exploring the concept of free-floating public displays , 2014, CHI Extended Abstracts.

[38]  I. Scott MacKenzie,et al.  The use of gaze to control drones , 2014, ETRA.

[39]  Bilge Mutlu,et al.  Communication of Intent in Assistive Free Flyers , 2014, 2014 9th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[40]  Koichi Hori,et al.  Entertainment Multi-rotor Robot that Realises Direct and Multimodal Interaction , 2014, BCS HCI.

[41]  Aníbal Ollero,et al.  Multimodal Interface Technologies for UAV Ground Control Stations , 2010, J. Intell. Robotic Syst..

[42]  Ana Paiva,et al.  Social Robots for Long-Term Interaction: A Survey , 2013, International Journal of Social Robotics.

[43]  Simon J. Julier,et al.  Towards a situated, multimodal interface for multiple UAV control , 2010, 2010 IEEE International Conference on Robotics and Automation.

[44]  S. Vignesh,et al.  UNMANNED FIRE EXTINGUISHER USING QUADCOPTER , 2017 .

[45]  Koichi Hori,et al.  Above your hand: direct and natural interaction with aerial robot , 2014, SIGGRAPH '14.

[46]  Pascal Mettes,et al.  Nature Conservation Drones for Automatic Localization and Counting of Animals , 2014, ECCV Workshops.

[47]  K. Lafleur,et al.  Quadcopter control in three-dimensional space using a noninvasive motor imagery-based brain–computer interface , 2013, Journal of neural engineering.

[48]  Walid Saad,et al.  Prospect theory for enhanced cyber-physical security of drone delivery systems: A network interdiction game , 2017, 2017 IEEE International Conference on Communications (ICC).

[49]  Jürgen Scheible,et al.  Displaydrone: a flying robot based interactive display , 2013, PerDis.

[50]  Franck Tarpin-Bernard,et al.  Towards Brain Computer Interfaces for Recreational Activities: Piloting a Drone , 2015, INTERACT.

[51]  Luca Maria Gambardella,et al.  Human Control of UAVs using Face Pose Estimates and Hand Gestures , 2014, 2014 9th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[52]  Geet Krishna Capoor,et al.  Gesture control of drone using a motion controller , 2016, 2016 International Conference on Industrial Informatics and Computer Systems (CIICS).

[53]  Markus Funk,et al.  DroneNavigator: Using Drones for Navigating Visually Impaired Persons , 2015, ASSETS.

[54]  Walid Saad,et al.  Unmanned Aerial Vehicle With Underlaid Device-to-Device Communications: Performance and Tradeoffs , 2015, IEEE Transactions on Wireless Communications.

[55]  James A. Landay,et al.  Emotion encoding in Human-Drone Interaction , 2016, 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[56]  Bilge Mutlu,et al.  Communicating Directionality in Flying Robots , 2015, 2015 10th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[57]  Hendrik Knoche,et al.  Knowing You, Seeing Me: Investigating User Preferences in Drone-Human Acknowledgement , 2018, CHI.

[58]  Jan Faigl,et al.  AR-Drone as a Platform for Robotic Research and Education , 2011, Eurobot Conference.

[59]  James A. Landay,et al.  Drone & me: an exploration into natural human-drone interaction , 2015, UbiComp.

[60]  James A. Landay,et al.  Drone & Wo: Cultural Influences on Human-Drone Interaction Techniques , 2017, CHI.

[61]  J.F.B. Coenen UAV BCI Comparison of manual and pedal control systems for 2D flight performance of users with simultaneous BCI control , 2015 .

[62]  Keum-Shik Hong,et al.  Hybrid EEG-NIRS based BCI for quadcopter control , 2015, 2015 54th Annual Conference of the Society of Instrument and Control Engineers of Japan (SICE).

[63]  Federico Manuri,et al.  A Kinect-based natural interface for quadrotor control , 2011, Entertain. Comput..

[64]  Ehud Sharlin,et al.  Flying Frustum: A Spatial Interface for Enhancing Human-UAV Awareness , 2015, HAI.

[65]  Markus Funk,et al.  In-situ-displaydrone: facilitating co-located interactive experiences via a flying screen , 2016, PerDis.

[66]  Minho Kim,et al.  Quadcopter flight control using a low-cost hybrid interface with EEG-based classification and eye tracking , 2014, Comput. Biol. Medicine.

[67]  Greg Mori,et al.  HRI in the sky: Creating and commanding teams of UAVs with a vision-mediated gestural interface , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[68]  Mason Chilmonczyk Kinect Control of a Quadrotor UAV , 2015 .