Real-time photo-realistic visualization of 3D environments for enhanced tele-operation of vehicles

This paper describes a method for creating photorealistic three-dimensional (3D) models of real-world environments in real-time for the purpose of improving and extending the capabilities of vehicle tele-operation. Our approach utilizes the combined data from a laser scanner (for modeling 3D geometry) and a video camera (for modeling surface appearance). The sensors are mounted on a moving vehicle platform, and a photo-realistic 3D model of the vehicle's environment is generated and displayed to the remote operator in real time. Our model consists of three main components: a textured ground surface, textured or colorized non-ground objects, and a textured background for representing regions beyond the laser scanner's sensing horizon. Our approach enables many unique capabilities for vehicle tele-operation, including viewing the scene from virtual viewpoints (e.g., behind the vehicle or top down), seamless augmentation of the environment with digital objects, and improved robustness to transmission latencies and data dropouts.

[1]  Aaron C. Courville,et al.  A Generative Model of Terrain for Autonomous Navigation in Vegetation , 2006, Int. J. Robotics Res..

[2]  Richard P. Paul,et al.  Teleprogramming: Toward Delay-Invariant Remote Manipulation , 1992, Presence: Teleoperators & Virtual Environments.

[3]  Terrence Fong,et al.  VEVI: A Virtual Environment Teleoperations Interface for Planetary Exploration , 1995 .

[4]  Lance Williams,et al.  Casting curved shadows on curved surfaces , 1978, SIGGRAPH.

[5]  Francois Blais,et al.  System for indoor 3D mapping and virtual environments , 1997, Optics & Photonics.

[6]  Ioannis Stamos,et al.  Geometry and Texture Recovery of Scenes of Large Scale , 2002, Comput. Vis. Image Underst..

[7]  Nghia Ho,et al.  Large Scale 3D Environmental Modelling for Stereoscopic Walk-Through Visualisation , 2007, 2007 3DTV Conference.

[8]  Leonard McMillan,et al.  Plenoptic Modeling: An Image-Based Rendering System , 2023 .

[9]  Klaus Diepold,et al.  Multi-modal multi-user telepresence and teleaction system , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[10]  Peter Rander,et al.  A Multi-Camera Method for 3D Digitization of Dynamic, Real-World Events , 1998 .

[11]  Robert Meyers,et al.  Real-time photorealistic virtualized reality interface for remote mobile robot control , 2010, ISRR.

[12]  Christian Früh,et al.  Data Processing Algorithms for Generating Textured 3D Building Facade Meshes from Laser Scans and Camera Images , 2005, International Journal of Computer Vision.

[13]  Joel Alberts,et al.  Manipulator Autonomy for EOD Robots , 2008 .

[14]  Takeo Kanade,et al.  Virtualized Reality: Constructing Virtual Worlds from Real Scenes , 1997, IEEE Multim..

[15]  Piotr Jasiobedzki,et al.  Stereo-Vision Based 3D Modeling and Localization for Unmanned Vehicles , 2008 .

[16]  Mark Segal,et al.  Fast shadows and lighting effects using texture mapping , 1992, SIGGRAPH.

[17]  Suya You,et al.  Approaches to Large-Scale Urban Modeling , 2003, IEEE Computer Graphics and Applications.

[18]  Alexei A. Efros,et al.  Geometric context from a single image , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[19]  Hans P. Moravec.  Robot spatial perception by stereoscopic vision and 3D evidence grids , 1996 .

[20]  Dinesh K. Pai,et al.  Model-based telerobotics with vision , 1997, Proceedings of International Conference on Robotics and Automation.

[21]  Sudipta N. Sinha,et al.  REAL-TIME VIDEO-BASED RECONSTRUCTION OF URBAN ENVIRONMENTS , 2007 .