A Comparison of Adaptive View Techniques for Exploratory 3D Drone Teleoperation

Drone navigation in complex environments poses many problems to teleoperators. Especially in three dimensional (3D) structures such as buildings or tunnels, viewpoints are often limited to the drone’s current camera view, nearby objects can be collision hazards, and frequent occlusion can hinder accurate manipulation. To address these issues, we have developed a novel interface for teleoperation that provides a user with environment-adaptive viewpoints that are automatically configured to improve safety and provide smooth operation. This real-time adaptive viewpoint system takes robot position, orientation, and 3D point-cloud information into account to modify the user’s viewpoint to maximize visibility. Our prototype uses simultaneous localization and mapping (SLAM) based reconstruction with an omnidirectional camera, and we use the resulting models as well as simulations in a series of preliminary experiments testing navigation of various structures. Results suggest that automatic viewpoint generation can outperform first- and third-person view interfaces for virtual teleoperators in terms of ease of control and accuracy of robot operation.

[1]  Masanao Koeda,et al.  Development of an immersive teleoperating system for unmanned helicopter , 2002, Proceedings. 11th IEEE International Workshop on Robot and Human Interactive Communication.

[2]  Daniel Cremers,et al.  LSD-SLAM: Large-Scale Direct Monocular SLAM , 2014, ECCV.

[3]  F. Matsuno,et al.  Study on effective camera images for mobile robot teleoperation , 2004, RO-MAN 2004. 13th IEEE International Workshop on Robot and Human Interactive Communication (IEEE Catalog No.04TH8759).

[4]  Steve Swink.  Super Mario 64 , 2008 .

[5]  Charles R. Dyer,et al.  Interpolating view and scene motion by dynamic view morphing , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[6]  Atsushi Yamashita,et al.  A decoupled virtual camera using spherical optical flow , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[7]  Naokazu Yokoya,et al.  [Paper] Free-viewpoint Mobile Robot Teleoperation Interface Using View-dependent Geometry and Texture , 2014 .

[8]  François Michaud,et al.  Egocentric and exocentric teleoperation interface using real-time, 3D video projection , 2009, 2009 4th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[9]  Kiyoshi Kiyokawa,et al.  Adaptive View Management for Drone Teleoperation in Complex 3D Structures , 2017, IUI.

[10]  Takeo Igarashi,et al.  A teleoperating interface for ground vehicles using autonomous flying cameras , 2013, 2013 23rd International Conference on Artificial Reality and Telexistence (ICAT).

[11]  Alex Zelinsky,et al.  Learning OpenCV---Computer Vision with the OpenCV Library (Bradski, G.R. et al.; 2008)[On the Shelf] , 2009, IEEE Robotics & Automation Magazine.

[12]  Ken Shoemake,et al.  Animating rotation with quaternion curves , 1985, SIGGRAPH.

[13]  David W. Hainsworth,et al.  Teleoperation User Interfaces for Mining Robotics , 2001, Auton. Robots.

[14]  Roland Siegwart,et al.  Vision-Controlled Micro Flying Robots: From System Design to Autonomous Navigation and Mapping in GPS-Denied Environments , 2014, IEEE Robotics & Automation Magazine.

[15]  Vijay Kumar,et al.  Autonomous Navigation and Mapping for Inspection of Penstocks and Tunnels With MAVs , 2017, IEEE Robotics and Automation Letters.

[16]  Dieter Schmalstieg,et al.  Image-driven view management for augmented reality browsers , 2012, 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[17]  Yuichi Ohta,et al.  NaviView: Virtual mirrors for visual assistance at blind intersection , 2005 .

[18]  Titus Cieslewski,et al.  Rapid exploration with multi-rotors: A frontier selection method for high speed flight , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[19]  Kiyoshi Kiyokawa,et al.  A 2D-3D integrated interface for mobile robot control using omnidirectional images and 3D geometric models , 2006, 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality.

[20]  Yasushi Yagi,et al.  Super Wide Viewing for Tele-operation , 2004 .

[21]  Stefanie Zollmann,et al.  FlyAR: Augmented Reality Supported Micro Aerial Vehicle Navigation , 2014, IEEE Transactions on Visualization and Computer Graphics.

[22]  Vijay Kumar,et al.  Inspection of Penstocks and Featureless Tunnel-like Environments Using Micro UAVs , 2013, FSR.

[23]  Jürgen Schmidhuber,et al.  A Machine Learning Approach to Visual Perception of Forest Trails for Mobile Robots , 2016, IEEE Robotics and Automation Letters.

[24]  Daniel Cremers,et al.  Camera-based navigation of a low-cost quadrocopter , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[25]  Lennart E. Nacke,et al.  More than a feeling: Measurement of sonic user experience and psychophysiology in a first-person shooter game , 2010, Interact. Comput..

[26]  Kiyoshi Kiyokawa,et al.  Dynamic text management for see-through wearable and heads-up display systems , 2013, IUI '13.

[27]  Greg Mori,et al.  HRI in the sky: Creating and commanding teams of UAVs with a vision-mediated gestural interface , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[28]  Kazuya Yoshida,et al.  Collaborative mapping of an earthquake‐damaged building via ground and aerial robots , 2012, J. Field Robotics.

[29]  Shoichi Maeyama,et al.  View Point Decision Algorithm for an Autonomous Robot to Provide Support Images in the Operability of a Teleoperated Robot , 2016 .

[30]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[31]  Yasushi Yagi,et al.  Omnidirectional imaging with hyperboloidal projection , 1993, Proceedings of 1993 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS '93).

[32]  Wagner James Au.  The Making of Second Life: Notes from the New World , 2008 .

[33]  Flavio Fontana,et al.  Autonomous, Vision‐based Flight and Live Dense 3D Mapping with a Quadrotor Micro Aerial Vehicle , 2016, J. Field Robotics.