A Practical Extrinsic calibration method for joint depth and color sensors

Abstract In recent years, with the development of navigation applications, depth and color sensor pairs have been widely used in many onboard systems of vision and robotics, for example, Automated Guided Vehicle (AGV). To effectively use the 3D data from the depth sensor and 2D data from the color sensor, extrinsic calibration is a fundamental problem. However, the existing calibration methods are mostly target-based or complicated, which could not fit onboard systems well. A flexible and practical approach is still expected. This paper presents an easy-to-use calibration method based on the line of two intersecting planes (e.g., floor and wall), which can be easily found in artificial environments. Make a planar motion of a set of depth and color sensors on the floor. Since the set's motion is relative to the line, we could consider the set static and the line to be moving with reference to the set. Lots of coplanar intersection points of the lines can be obtained from the depth sensor (3D) and color sensor (2D) together. Using these 3D-2D corresponding point pairs, a homography relationship about the floor between the depth sensor and color sensor can be computed. The extrinsic parameters between these two sensors can be solved in a closed-form. Our method is verified on both simulated and real data, and the experiments show that the method can perform well in terms of robustness and accuracy.

[1]  Patrick Rives,et al.  Extrinsic Calibration of Multiple RGB-D Cameras From Line Observations , 2018, IEEE Robotics and Automation Letters.

[2]  Kwang In Kim,et al.  RGBD-Dog: Predicting Canine Pose from RGBD Sensors , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Tomás Pajdla,et al.  3D with Kinect , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[4]  Eung-Su Kim,et al.  Extrinsic Calibration between Camera and LiDAR Sensors by Matching Multiple 3D Planes † , 2019, Sensors.

[5]  Aggelos K. Katsaggelos,et al.  Automatic, fast, online calibration between depth and color cameras , 2014, J. Vis. Commun. Image Represent..

[6]  Juan I. Nieto,et al.  Motion-Based Calibration of Multimodal Sensor Extrinsics and Timing Offset Estimation , 2016, IEEE Transactions on Robotics.

[7]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Elise Lachat,et al.  Assessment and Calibration of a RGB-D Camera (Kinect v2 Sensor) Towards a Potential Use for Close-Range 3D Modeling , 2015, Remote. Sens..

[9]  Yi Ma,et al.  End-to-End Wireframe Parsing , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[10]  Zoltan Kato,et al.  Targetless Calibration of a Lidar - Perspective Camera Pair , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[11]  Hans-Joachim Wünsche,et al.  Odometry-based online extrinsic sensor calibration , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[12]  Silvio Savarese,et al.  Extrinsic Calibration of a 3D Laser Scanner and an Omnidirectional Camera , 2010 .

[13]  Seon-Min Rhee,et al.  Time-of-flight sensor and color camera calibration for multi-view acquisition , 2011, The Visual Computer.

[14]  Roland Siegwart,et al.  Extrinsic self calibration of a camera and a 3D laser range finder from natural scenes , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[15]  Zhengyou Zhang,et al.  Calibration between depth and color sensors for commodity depth cameras , 2011, 2011 IEEE International Conference on Multimedia and Expo.

[16]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[17]  Yangyu Fan,et al.  A new method for calibrating depth and color camera pair based on Kinect , 2012, 2012 International Conference on Audio, Language and Image Processing.

[18]  Zhenyu Guo,et al.  RGGNet: Tolerance Aware LiDAR-Camera Online Calibration With Geometric Deep Learning and Generative Model , 2020, IEEE Robotics and Automation Letters.

[19]  Zhuo Qing,et al.  A New Approach to Calibrate Range Image and Color Image from Kinect , 2012, 2012 4th International Conference on Intelligent Human-Machine Systems and Cybernetics.

[20]  Honghai Liu,et al.  Simultaneous Calibration: A Joint Optimization Approach for Multiple Kinect and External Cameras , 2017, Sensors.

[21]  Yang Cao,et al.  Contrast Prior and Fluid Pyramid Integration for RGBD Salient Object Detection , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Urbano Nunes,et al.  Fast and Accurate Calibration of a Kinect Sensor , 2013, 2013 International Conference on 3D Vision.

[23]  Vincent Frémont,et al.  Circular Targets for 3D Alignment of Video and Lidar Sensors , 2012, Adv. Robotics.

[24]  J. Dennis,et al.  Derivative free analogues of the Levenberg-Marquardt and Gauss algorithms for nonlinear least squares approximation , 1971 .

[25]  E. K. Forkuo.  Automatic fusion of photogrammetric imagery and laser scanner point clouds , 2003 .

[26]  Michael S. Brown,et al.  High quality depth map upsampling for 3D-TOF cameras , 2011, 2011 International Conference on Computer Vision.

[27]  Ju Shen,et al.  Extrinsic calibration for wide-baseline RGB-D camera network , 2014, 2014 IEEE 16th International Workshop on Multimedia Signal Processing (MMSP).

[28]  Ju Shen,et al.  A Fast and Robust Extrinsic Calibration for RGB-D Camera Networks † , 2018, Sensors.

[29]  Nanning Zheng,et al.  Line Feature Based Extrinsic Calibration of LiDAR and Camera , 2018, 2018 IEEE International Conference on Vehicular Electronics and Safety (ICVES).

[30]  Radu Horaud,et al.  An analytic solution for the perspective 4-point problem , 1989, Proceedings CVPR '89: IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[31]  Shenghua Gao,et al.  PPGNet: Learning Point-Pair Graph for Line Segment Detection , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Rafael Grompone von Gioi,et al.  LSD: A Fast Line Segment Detector with a False Detection Control , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Juho Kannala,et al.  Accurate and Practical Calibration of a Depth and Color Camera Pair , 2011, CAIP.

[34]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[35]  Silvio Savarese,et al.  Automatic Extrinsic Calibration of Vision and Lidar by Maximizing Mutual Information , 2015, J. Field Robotics.

[36]  Stewart Worrall,et al.  Automatic extrinsic calibration between a camera and a 3D Lidar using 3D point and plane correspondences , 2019, 2019 IEEE Intelligent Transportation Systems Conference (ITSC).

[37]  Stanley M. Bileschi,et al.  Fully automatic calibration of LIDAR and video streams from a vehicle , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[38]  Juho Kannala,et al.  Joint Depth and Color Camera Calibration with Distortion Correction , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[39]  Urbano Nunes,et al.  A Minimal Solution for the Extrinsic Calibration of a Camera and a Laser-Rangefinder , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Jun Luo,et al.  A Novel Method for Extrinsic Calibration of Multiple RGB-D Cameras Using Descriptor-Based Patterns , 2019, Sensors.

[41]  In-So Kweon,et al.  Time-of-Flight Sensor Calibration for a Color and Depth Camera Pair , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Dana H. Ballard,et al.  Generalizing the Hough transform to detect arbitrary shapes , 1981, Pattern Recognit..

[43]  Michael Bosse,et al.  Line-based extrinsic calibration of range and image sensors , 2013, 2013 IEEE International Conference on Robotics and Automation.

[44]  Hongdong Li,et al.  Accurate extrinsic calibration between monocular camera and sparse 3D Lidar points without markers , 2017, 2017 IEEE Intelligent Vehicles Symposium (IV).

[45]  Sebastian Thrun,et al.  Automatic Online Calibration of Cameras and Lasers , 2013, Robotics: Science and Systems.