A Deep-Learning Based Multi-Modality Sensor Calibration Method for USV

The automatic obstacle avoidance and other tasks of the unmanned surface vehicle rely on the fusion of multi-modality onboard sensors. The accurate calibration method is the foundation of sensor fusion. This paper proposes an online calibration method based on the deep learning for visual sensor and depth sensor. Through an end-to-end network, we combine feature extraction, feature matching and global optimization process of sensor calibration. After initial training, the network can continuously calibrate multi-modality sensors. It solves the USV calibration challenges under complex operating environment. In the simulation environment and realistic environment, we conducted a fast online calibration of the camera, LIDAR and depth camera, which showed the effectiveness of the algorithm.

[1]  Nick Schneider,et al.  RegNet: Multimodal sensor registration using deep neural networks , 2017, 2017 IEEE Intelligent Vehicles Symposium (IV).

[2]  Keiji Nagatani,et al.  Field report: Autonomous lake bed depth mapping by a portable semi-submersible USV at Mt. Zao Okama Crater lake , 2016, 2016 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR).

[3]  Jason Gu,et al.  The obstacle detection and obstacle avoidance algorithm based on 2-D lidar , 2015, 2015 IEEE International Conference on Information and Automation.

[4]  Roberto Cipolla,et al.  PoseNet: A Convolutional Network for Real-Time 6-DOF Camera Relocalization , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[5]  Qixiang Ye,et al.  Combined feature evaluation for adaptive visual object tracking , 2011, Comput. Vis. Image Underst..

[6]  Robin R. Murphy,et al.  UAV assisted USV visual navigation for marine mass casualty incident response , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[7]  Enrico Simetti,et al.  Towards the Use of a Team of USVs for Civilian Harbour Protection: USV Interception of Detected Menaces , 2010 .

[8]  Michael Giering,et al.  Multi-modal sensor registration for vehicle perception via deep neural networks , 2014, 2015 IEEE High Performance Extreme Computing Conference (HPEC).

[9]  Joachim L. Grenestedt,et al.  LORCA: A high performance USV with applications to surveillance and monitoring , 2015, 2015 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR).

[10]  Carlos Hernandez,et al.  Multi-View Stereo: A Tutorial , 2015, Found. Trends Comput. Graph. Vis..

[11]  Avinash C. Kak,et al.  Vision-based navigation by a mobile robot with obstacle avoidance using single-camera vision and ultrasonic sensing , 1998, IEEE Trans. Robotics Autom..

[12]  Hu Zhan.  A Review on Some Active Vision Based Camera Calibration Techniques , 2002 .

[13]  Gang Xu,et al.  Epipolar Geometry in Stereo, Motion and Object Recognition , 1996, Computational Imaging and Vision.

[14]  Sei Ikeda,et al.  Visual SLAM algorithms: a survey from 2010 to 2016 , 2017, IPSJ Transactions on Computer Vision and Applications.

[15]  Levente Hajder,et al.  Accurate Calibration of LiDAR-Camera Systems Using Ordinary Boxes , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[16]  Stanley M. Bileschi,et al.  Fully automatic calibration of LIDAR and video streams from a vehicle , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[17]  Esa Rahtu,et al.  Relative Camera Pose Estimation Using Convolutional Neural Networks , 2017, ACIVS.

[18]  Qiang Chen,et al.  Network In Network , 2013, ICLR.

[19]  Dimitrios G. Kottas,et al.  3D LIDAR–camera intrinsic and extrinsic calibration: Identifiability and analytical least-squares-based initialization , 2012, Int. J. Robotics Res..

[20]  Wang Qi,et al.  Review on camera calibration , 2010, 2010 Chinese Control and Decision Conference.

[21]  Sebastian Thrun,et al.  Automatic Online Calibration of Cameras and Lasers , 2013, Robotics: Science and Systems.

[22]  Thomas Brox,et al.  FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Thomas Brox,et al.  FlowNet: Learning Optical Flow with Convolutional Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[24]  Robert Sutton,et al.  Advances in Unmanned Marine Vehicles , 2006 .

[25]  Robin R. Murphy,et al.  Visual pose estimation of USV from UAV to assist drowning victims recovery , 2016, 2016 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR).

[26]  Silvio Savarese,et al.  Automatic Extrinsic Calibration of Vision and Lidar by Maximizing Mutual Information , 2015, J. Field Robotics.

[27]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..