Generation and visualization of large‐scale three‐dimensional reconstructions from underwater robotic surveys

Robust, scalable simultaneous localization and mapping (SLAM) algorithms support the successful deployment of robots in real‐world applications. In many cases these platforms deliver vast amounts of sensor data from large‐scale, unstructured environments. These data may be difficult to interpret by end users without further processing and suitable visualization tools. We present a robust, automated system for large‐scale three‐dimensional (3D) reconstruction and visualization that takes stereo imagery from an autonomous underwater vehicle (AUV) and SLAM‐based vehicle poses to deliver detailed 3D models of the seafloor in the form of textured polygonal meshes. Our system must cope with thousands of images, lighting conditions that create visual seams when texturing, and possible inconsistencies between stereo meshes arising from errors in calibration, triangulation, and navigation. Our approach breaks down the problem into manageable stages by first estimating local structure and then combining these estimates to recover a composite georeferenced structure using SLAM‐based vehicle pose estimates. A texture‐mapped surface at multiple scales is then generated that is interactively presented to the user through a visualization engine. We adapt established solutions when possible, with an emphasis on quickly delivering approximate yet visually consistent reconstructions on standard computing hardware. This allows scientists on a research cruise to use our system to design follow‐up deployments of the AUV and complementary instruments. To date, this system has been tested on several research cruises in Australian waters and has been used to reliably generate and visualize reconstructions for more than 60 dives covering diverse habitats and representing hundreds of linear kilometers of survey. © 2009 Wiley Periodicals, Inc.

[1]  Martin Kraus,et al.  Pyramid Methods in GPU-Based Image Processing , 2011 .

[2]  Maria Byrne.  Flashing stars light up the Reef's shelf , 2009 .

[3]  Oscar Pizarro,et al.  Towards large scale optical and acoustic sensor integration for Visualization , 2009, OCEANS 2009-EUROPE.

[4]  Stefan B. Williams,et al.  AUV Benthic Habitat Mapping in South Eastern Tasmania , 2009, FSR.

[5]  Gertjan J. Burghouts,et al.  Performance evaluation of local colour invariants , 2009, Comput. Vis. Image Underst..

[6]  Daniel M. Helmick,et al.  Autonomy for Mars Rovers: Past, Present, and Future , 2008, Computer.

[7]  Stefan B. Williams,et al.  Efficient View-Based SLAM Using Visual Loop Closures , 2008, IEEE Transactions on Robotics.

[8]  Andrew Hogue,et al.  Modeling Underwater Structures , 2008, Int. J. Cogn. Informatics Nat. Intell..

[9]  M. Johnson-Roberson,et al.  AUV-assisted surveying of relic reef sites , 2008, OCEANS 2008.

[10]  Luc Van Gool,et al.  Fast scale invariant feature detection and matching on programmable graphics hardware , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[11]  Stefan B. Williams,et al.  From Corals to Canyons: The Great Barrier Reef Margin , 2008 .

[12]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[13]  Franz S. Hover,et al.  SLAM for ship hull inspection using exactly sparse extended information filters , 2008, 2008 IEEE International Conference on Robotics and Automation.

[14]  Garrison W. Cottrell,et al.  Gamma-SLAM: Using stereo vision and variance grid maps for SLAM in unstructured environments , 2008, 2008 IEEE International Conference on Robotics and Automation.

[15]  T. Nicosevici,et al.  Online Robust 3D Mapping Using Structure from Motion Cues , 2008, OCEANS 2008 - MTS/IEEE Kobe Techno-Ocean.

[16]  C. Langmuir,et al.  Hydrothermal exploration with the Autonomous Benthic Explorer , 2008 .

[17]  Jitendra Malik,et al.  Recovering high dynamic range radiance maps from photographs , 1997, SIGGRAPH '08.

[18]  I. Mahon.  Vision-based navigation for autonomous underwater vehicles , 2008 .

[19]  Ioannis M. Rekleitis,et al.  Over-the-horizon, autonomous navigation for planetary exploration , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[20]  Wolfram Burgard,et al.  Learning maps in 3D using attitude and noisy vision sensors , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[21]  Jan-Michael Frahm,et al.  Detailed Real-Time Urban 3D Reconstruction from Video , 2007, International Journal of Computer Vision.

[22]  Jie Wu,et al.  A New Progressive Mesh with Adaptive Subdivision for LOD Models , 2007, ICEC.

[23]  Simon Lacroix,et al.  Vision-Based SLAM: Stereo and Monocular Approaches , 2007, International Journal of Computer Vision.

[24]  Nghia Ho,et al.  Large Scale 3D Environmental Modelling for Stereoscopic Walk-Through Visualisation , 2007, 2007 3DTV Conference.

[25]  Robert C. Bolles,et al.  Localization and Mapping for Autonomous Navigation in Outdoor Terrains : A Stereo Vision Approach , 2007, 2007 IEEE Workshop on Applications of Computer Vision (WACV '07).

[26]  Hanumant Singh,et al.  Towards High-resolution Imaging from Underwater Vehicles , 2007, Int. J. Robotics Res..

[27]  Hanumant Singh,et al.  A Self‐Consistent Bathymetric Mapping Algorithm , 2007, J. Field Robotics.

[28]  Jan-Michael Frahm,et al.  Feature tracking and matching in video using programmable graphics hardware , 2007, Machine Vision and Applications.

[29]  Hanumant Singh,et al.  Visually Mapping the RMS Titanic: Conservative Covariance Estimates for SLAM Information Filters , 2006, Int. J. Robotics Res..

[30]  Carlos Granell,et al.  Continuous Level of Detail on Graphics Hardware , 2006, DGCI.

[31]  Sebastian Thrun,et al.  Stanley: The robot that won the DARPA Grand Challenge , 2006, J. Field Robotics.

[32]  Hugh Durrant-Whyte,et al.  Simultaneous localization and mapping (SLAM): part II , 2006 .

[33]  P. Firoozfam,et al.  An ROV Stereovision System for Ship-Hull Inspection , 2006, IEEE Journal of Oceanic Engineering.

[34]  Michael M. Kazhdan,et al.  Poisson surface reconstruction , 2006, SGP '06.

[35]  Richard Szeliski,et al.  A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[36]  Hugh F. Durrant-Whyte,et al.  Simultaneous localization and mapping: part I , 2006, IEEE Robotics & Automation Magazine.

[37]  Andrew Hogue,et al.  Underwater 3D SLAM through entropy minimization , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[38]  C. Roman,et al.  Characterizing the deep insular shelf coral reef habitat of the Hind Bank marine conservation district (US Virgin Islands) using the Seabed autonomous underwater vehicle , 2006 .

[39]  Brian Bingham,et al.  Techniques for Deep Sea Near Bottom Survey Using an Autonomous Underwater Vehicle , 2007, Int. J. Robotics Res..

[40]  Michael M. Kazhdan,et al.  Reconstruction of solid models from oriented point sets , 2005, SGP '05.

[41]  Dana R. Yoerger,et al.  A Serpentinite-Hosted Ecosystem: The Lost City Hydrothermal Field , 2005, Science.

[42]  P. J. Huber.  Robust Statistics: Huber/Robust Statistics , 2005 .

[43]  Wolfram Burgard,et al.  Autonomous exploration and mapping of abandoned mines , 2004, IEEE Robotics & Automation Magazine.

[44]  R. Eustice,et al.  Large area 3D reconstructions from underwater surveys , 2004, Oceans '04 MTS/IEEE Techno-Ocean '04 (IEEE Cat. No.04CH37600).

[45]  H. Singh,et al.  Advances in large-area photomosaicking underwater , 2004, IEEE Journal of Oceanic Engineering.

[46]  David G. Lowe,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004, International Journal of Computer Vision.

[47]  Christian Früh,et al.  An Automated Method for Large-Scale, Ground-Based City Model Acquisition , 2004, International Journal of Computer Vision.

[48]  S. Sukkarieh,et al.  Autonomous airborne navigation in unknown terrain environments , 2004, IEEE Transactions on Aerospace and Electronic Systems.

[49]  Stefan B. Williams,et al.  Simultaneous localisation and mapping on the Great Barrier Reef , 2004, IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004.

[50]  C. Roman,et al.  Seabed AUV offers new platform for high‐resolution imaging , 2004 .

[51]  Takeo Kanade,et al.  Shape and motion from image streams under orthography: a factorization method , 1992, International Journal of Computer Vision.

[52]  Mark Segal,et al.  The OpenGL Graphics System: A Specification , 2004 .

[53]  C. Roman,et al.  Imaging Coral I: Imaging Coral Habitats with the SeaBED AUV , 2004 .

[54]  Suya You,et al.  Approaches to Large-Scale Urban Modeling , 2003, IEEE Computer Graphics and Applications.

[55]  Hanumant Singh,et al.  Toward large-area mosaicing for underwater scientific applications , 2003 .

[56]  Shahriar Negahdaripour,et al.  Stereovision imaging on submersible platforms for 3-D mapping of benthic habitats and sea-floor structures , 2003 .

[57]  Hans-Peter Seidel,et al.  Multi-level partition of unity implicits , 2003, ACM Trans. Graph..

[58]  X. Cufi,et al.  On the way to solve lighting problems in underwater imaging , 2002, OCEANS '02 MTS/IEEE.

[59]  Brian V. Funt,et al.  A comparison of computational color constancy algorithms. I: Methodology and experiments with synthesized data , 2002, IEEE Trans. Image Process..

[60]  Xun Xu,et al.  Mosaic-based positioning and improved motion-estimation methods for automatic navigation of submersible vehicles , 2002 .

[61]  D. Scharstein,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[62]  Richard Szeliski,et al.  Eliminating ghosting and exposure artifacts in image mosaics , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[63]  J. Santos-Victor,et al.  Underwater mosaicing and trajectory reconstruction using global alignment , 2001, MTS/IEEE Oceans 2001. An Ocean Odyssey. Conference Proceedings (IEEE Cat. No.01CH37295).

[64]  Hugh F. Durrant-Whyte,et al.  A solution to the simultaneous localization and map building (SLAM) problem , 2001, IEEE Trans. Robotics Autom..

[65]  Julie Dorsey,et al.  Reconstructing 3D Tree Models from Instrumented Photographs , 2001, IEEE Computer Graphics and Applications.

[66]  Patrick J. Flynn,et al.  A Survey Of Free-Form Object Representation and Recognition Techniques , 2001, Comput. Vis. Image Underst..

[67]  Niels Jørgen Christensen,et al.  Fast Texture-Based Form Factor Calculations for Radiosity Using Graphics Hardware , 2001, J. Graphics, GPU, & Game Tools.

[68]  Reinhard Koch,et al.  Automated reconstruction of 3D scenes from sequences of images , 2000 .

[69]  D. Yoerger,et al.  The discovery of ancient history in the deep sea using advanced deep submergence technology , 2000 .

[70]  Marc Levoy,et al.  The digital Michelangelo project: 3D scanning of large statues , 2000, SIGGRAPH.

[71]  Jonathan C. Howland.  Digital Data Logging and Processing, Derbyshire Survey, 1997 , 1999 .

[72]  Shahriar Negahdaripour,et al.  Direct estimation of motion from sea floor images for automatic station-keeping of submersible platforms , 1999 .

[73]  Michael Garland,et al.  Simplifying surfaces with color and texture using quadric error metrics , 1998, Proceedings Visualization '98 (Cat. No.98CB36276).

[74]  Lance Williams.  Pyramidal parametrics , 1998 .

[75]  Marshall W. Bern,et al.  Surface Reconstruction by Voronoi Filtering , 1998, SCG '98.

[76]  Harpreet S. Sawhney,et al.  Robust Video Mosaicing through Topology Inference and Local to Global Alignment , 1998, ECCV.

[77]  Andrew W. Fitzgibbon,et al.  Automatic Camera Recovery for Closed or Open Image Sequences , 1998, ECCV.

[78]  Brian E. Smits.  Efficiency issues for ray tracing , 1998, J. Graphics, GPU, & Game Tools.

[79]  Michael Garland,et al.  Surface simplification using quadric error metrics , 1997, SIGGRAPH.

[80]  Harpreet S. Sawhney,et al.  True multi-image alignment and its application to mosaicing and lens distortion correction , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[81]  Marc Levoy,et al.  A volumetric method for building complex models from range images , 1996, SIGGRAPH.

[82]  H. H. Wang,et al.  Video mosaicking along arbitrary vehicle paths , 1996, Proceedings of Symposium on Autonomous Underwater Vehicle Technology.

[83]  Hugh F. Durrant-Whyte,et al.  An Autonomous Guided Vehicle for Cargo Handling Applications , 1995, Int. J. Robotics Res..

[84]  Karel J. Zuiderveld,et al.  Contrast Limited Adaptive Histogram Equalization , 1994, Graphics Gems.

[85]  Marc Levoy,et al.  Zippered polygon meshes from range images , 1994, SIGGRAPH.

[86]  Jules S. Jaffe,et al.  Computer modeling and the design of optimal underwater imaging systems , 1990 .

[87]  Christopher G. Harris,et al.  A Combined Corner and Edge Detector , 1988, Alvey Vision Conference.

[88]  Larry H. Matthies,et al.  Error modeling in stereo navigation , 1986, IEEE J. Robotics Autom..

[89]  Paul S. Heckbert,et al.  Survey of Texture Mapping , 1986, IEEE Computer Graphics and Applications.

[90]  Jean-Daniel Boissonnat,et al.  Geometric structures for three-dimensional shape representation , 1984, TOGS.

[91]  Edward H. Adelson,et al.  A multiresolution spline with application to image mosaics , 1983, TOGS.

[92]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[93]  James J. Little,et al.  Automatic extraction of Irregular Network digital terrain models , 1979, SIGGRAPH.

[94]  James H. Clark,et al.  Hierarchical geometric models for visible surface algorithms , 1976, CACM.

[95]  Edwin Earl Catmull,et al.  A subdivision algorithm for computer display of curved surfaces. , 1974 .

[96]  S. Duntley.  Light in the Sea , 1963 .