Underwater Depth Estimation for Spherical Images

This paper proposes a method for monocular underwater depth estimation, which is an open problem in robotics and computer vision. To this end, we leverage publicly available in-air RGB-D image pairs for underwater depth estimation in the spherical domain with an unsupervised approach. For this, the in-air images are style-transferred to the underwater style as the first step. Given those synthetic underwater images and their ground truth depth, we then train a network to estimate the depth. This way, our learning model is designed to obtain the depth up to scale, without the need of corresponding ground truth underwater depth data, which is typically not available. We test our approach on style-transferred in-air images as well as on our own real underwater dataset, for which we computed sparse ground truth depths data via stereopsis. This dataset is provided for download. Experiments with this data against a state-of-the-art in-air network as well as different artificial inputs show that the style transfer as well as the depth estimation exhibit promising performance.

[1]  Andreas Birk,et al.  Underwater image haze removal with an underwater-ready dark channel prior , 2017, OCEANS 2017 – Anchorage.

[2]  Nassir Navab,et al.  Distortion-Aware Convolutional Filters for Dense Prediction in Panoramic Images , 2018, ECCV.

[3]  Trevor Darrell,et al.  Adversarial Feature Learning , 2016, ICLR.

[4]  Jun Zhang,et al.  Underwater image dehaze using scene depth estimation with adaptive color correction , 2017, OCEANS 2017 - Aberdeen.

[5]  Sergio Guadarrama,et al.  Tracking Emerges by Colorizing Videos , 2018, ECCV.

[6]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[7]  Nassir Navab,et al.  Deeper Depth Prediction with Fully Convolutional Residual Networks , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[8]  Silvio Savarese,et al.  Joint 2D-3D-Semantic Data for Indoor Scene Understanding , 2017, ArXiv.

[9]  Qingwen Xu,et al.  Depth Estimation on Underwater Omni-directional Images Using a Deep Neural Network , 2019, ICRA 2019.

[10]  Sergey Levine,et al.  Grasp2Vec: Learning Object Representations from Self-Supervised Grasping , 2018, CoRL.

[11]  Matthew Johnson-Roberson,et al.  Automatic color correction for 3D reconstruction of underwater scenes , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[12]  Maide Bucolo,et al.  Computational models in microfluidic bubble logic , 2014, Microfluidics and Nanofluidics.

[13]  Noah Snavely,et al.  Unsupervised Learning of Depth and Ego-Motion from Video , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Pamela C. Cosman,et al.  Single underwater image enhancement using depth estimation based on blurriness , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[15]  Andreas Birk,et al.  Maximum likelihood mapping with spectral image registration , 2010, 2010 IEEE International Conference on Robotics and Automation.

[16]  Michael J. Black,et al.  Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Kuk-Jin Yoon,et al.  SpherePHD: Applying CNNs on a Spherical PolyHeDron Representation of 360° Images , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Antonis A. Argyros,et al.  Robot Homing by Exploiting Panoramic Vision , 2005, Auton. Robots.

[19]  Gabriel J. Brostow,et al.  Digging Into Self-Supervised Monocular Depth Estimation , 2018, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[20]  Ian D. Reid,et al.  Self-supervised Learning for Single View Depth and Surface Normal Estimation , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[21]  Silvia Silva da Costa Botelho,et al.  Transmission Estimation in Underwater Single Images , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[22]  Arturo Gomez Chavez,et al.  Adaptive Navigation Scheme for Optimal Deep-Sea Localization Using Multimodal Perception Cues , 2019, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[23]  Oisin Mac Aodha,et al.  Unsupervised Monocular Depth Estimation with Left-Right Consistency , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Stefano Soatto,et al.  Bilateral Cyclic Constraint and Adaptive Regularization for Unsupervised Monocular Depth Prediction , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Alexei A. Efros,et al.  Colorful Image Colorization , 2016, ECCV.

[26]  Sertac Karaman,et al.  Self-Supervised Sparse-to-Dense: Self-Supervised Depth Completion from LiDAR and Monocular Camera , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[27]  Pere Ridao,et al.  Omnidirectional Underwater Camera Design and Calibration , 2015, Sensors.

[28]  Matthew Johnson-Roberson,et al.  UWStereoNet: Unsupervised Learning for Depth Estimation and Color Correction of Underwater Stereo Imagery , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[29]  Pamela C. Cosman,et al.  Underwater Image Restoration Based on Image Blurriness and Light Absorption , 2017, IEEE Transactions on Image Processing.

[30]  Shenghua Gao,et al.  Saliency Detection in 360 ◦ Videos , 2022 .

[31]  Simon Lacroix,et al.  SLAM with Panoramic Vision , 2007, J. Field Robotics.

[32]  Arturo Gomez Chavez,et al.  Improved Fourier Mellin Invariant for Robust Rotation Estimation with Omni-Cameras , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[33]  Shugong Xu,et al.  Geometric Structure Based and Regularized Depth Estimation From 360 Indoor Imagery , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Frank Dellaert,et al.  3D reconstruction of underwater structures , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[35]  Thomas Brox,et al.  Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Jules S. Jaffe,et al.  Computer modeling and the design of optimal underwater imaging systems , 1990 .

[37]  Alexander H. Liu,et al.  Towards Scene Understanding: Unsupervised Monocular Depth Estimation With Semantic-Aware Representation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Ian D. Reid,et al.  Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[39]  Zhichao Yin,et al.  GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[40]  Sergey Levine,et al.  Contextual Imagined Goals for Self-Supervised Robotic Learning , 2019, CoRL.

[41]  László Neumann,et al.  Color transfer for underwater dehazing and depth estimation , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[42]  Nitish Srivastava.  Unsupervised Learning of Visual Representations using Videos , 2015 .

[43]  Rob Fergus,et al.  Depth Map Prediction from a Single Image using a Multi-Scale Deep Network , 2014, NIPS.

[44]  Sandro Barone,et al.  Experimentation of structured light and stereo vision for underwater 3D reconstruction , 2011 .

[45]  Frederik Schaffalitzky,et al.  A minimal solution for relative pose with unknown focal length , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[46]  Xuming He,et al.  Learning Autonomous Exploration and Mapping with Semantic Vision , 2019, Proceedings of the 2019 International Conference on Image, Video and Signal Processing - IVSP 2019.

[47]  Terry Boult,et al.  DOVE: Dolphin Omni-directional Video Equipment , 2000 .

[48]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[49]  Kaushik Mitra,et al.  Unsupervised Single Image Underwater Depth Estimation , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[50]  Jian Sun,et al.  Single image haze removal using dark channel prior , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[51]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[52]  Chamara Saroj Weerasekera,et al.  Visual Odometry Revisited: What Should Be Learnt? , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[53]  M. Bucolo,et al.  Nonlinear systems synchronization for modeling two-phase microfluidics flows , 2018 .

[54]  Mario Fernando Montenegro Campos,et al.  Underwater Depth Estimation and Image Restoration Based on Single Images , 2016, IEEE Computer Graphics and Applications.

[55]  Alexei A. Efros,et al.  Unsupervised Visual Representation Learning by Context Prediction , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[56]  Nikos Komodakis,et al.  Unsupervised Representation Learning by Predicting Image Rotations , 2018, ICLR.

[57]  Jie Li,et al.  WaterGAN: Unsupervised Generative Network to Enable Real-Time Color Correction of Monocular Underwater Images , 2017, IEEE Robotics and Automation Letters.

[58]  B. McGlamery.  ANALYSIS AND SIMULATION OF UNDERWATER CAMERA SYSTEM PERFORMANCE , 2008 .

[59]  Qingwen Xu,et al.  Pose Estimation for Omni-directional Cameras using Sinusoid Fitting , 2019, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).