Automatic Extrinsic Calibration of Vision and Lidar by Maximizing Mutual Information

This paper reports on an algorithm for automatic, targetless, extrinsic calibration of a lidar and optical camera system based upon the maximization of mutual information between the sensor-measured surface intensities. The proposed method is completely data-driven and does not require any fiducial calibration targets-making in situ calibration easy. We calculate the Cramer-Rao lower bound CRLB of the estimated calibration parameter variance, and we show experimentally that the sample variance of the estimated parameters empirically approaches the CRLB when the amount of data used for calibration is sufficiently large. Furthermore, we compare the calibration results to independent ground-truth where available and observe that the mean error empirically approaches zero as the amount of data used for calibration is increased, thereby suggesting that the proposed estimator is a minimum variance unbiased estimate of the calibration parameters. Experimental results are presented for three different lidar-camera systems: i a three-dimensional 3D lidar and omnidirectional camera, ii a 3D time-of-flight sensor and monocular camera, and iii a 2D lidar and monocular camera.

[1]  E. L. Lehmann,et al.  Theory of point estimation , 1950 .

[2]  Paul A. Viola,et al.  Alignment by Maximization of Mutual Information , 1997, International Journal of Computer Vision.

[3]  Korbinian Strimmer,et al.  Entropy Inference and the James-Stein Estimator, with Application to Nonlinear Gene Association Networks , 2008, J. Mach. Learn. Res..

[4]  Silvio Savarese,et al.  Visually bootstrapped generalized ICP , 2011, 2011 IEEE International Conference on Robotics and Automation.

[5]  Peter I. Corke,et al.  Cross-calibration of push-broom 2D LIDARs and cameras in natural scenes , 2013, 2013 IEEE International Conference on Robotics and Automation.

[6]  John A. Nelder,et al.  A Simplex Method for Function Minimization , 1965, Comput. J..

[7]  Bernard W. Silverman,et al.  Density Estimation for Statistics and Data Analysis , 1987 .

[8]  Juan I. Nieto,et al.  A mutual information approach to automatic calibration of camera and lidar in natural environments , 2012 .

[9]  Sebastian Thrun,et al.  Unsupervised Calibration for Multi-beam Lasers , 2010, ISER.

[10]  Alois Knoll,et al.  Mutual Information-Based 3D Object Tracking , 2008, International Journal of Computer Vision.

[11]  Dimitrios G. Kottas,et al.  3 D Lidar-Camera Intrinsic and Extrinsic Calibration : Observability Analysis and Analytical Least Squares-based Initialization , 2011 .

[12]  Wolfgang Wenzel,et al.  A stochastic tunneling approach for global minimization , 1999 .

[13]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[14]  S Forrest,et al.  Genetic algorithms , 1996, CSUR.

[15]  George Robinson,et al.  The Calculus of Observations - A Treatise on Numerical Mathematics , 1924 .

[16]  W. Wenzel,et al.  Stochastic Tunneling Approach for Global Minimization of Complex Potential Energy Landscapes , 1999 .

[17]  Rui P. Rocha,et al.  Data Fusion Calibration for a 3D Laser Range Finder and a Camera using Inertial Data , 2009, ECMR.

[18]  Anselmo Lastra,et al.  Automatic image alignment for 3D environment modeling , 2004, Proceedings. 17th Brazilian Symposium on Computer Graphics and Image Processing.

[19]  J. Mazziotta,et al.  MRI‐PET Registration with Automated Algorithm , 1993, Journal of computer assisted tomography.

[20]  Patrick Rives,et al.  Calibration between a central catadioptric camera and a laser range finder for robotic applications , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[21]  J. Borwein,et al.  Two-Point Step Size Gradient Methods , 1988 .

[22]  Sebastian Thrun,et al.  Automatic Online Calibration of Cameras and Lasers , 2013, Robotics: Science and Systems.

[23]  A. Chao,et al.  Nonparametric estimation of Shannon’s index of diversity when there are unseen species in sample , 2004, Environmental and Ecological Statistics.

[24]  Kenneth Levenberg.  A METHOD FOR THE SOLUTION OF CERTAIN NON – LINEAR PROBLEMS IN LEAST SQUARES , 1944 .

[25]  Ying Lin,et al.  3D LIDAR-Camera Extrinsic Calibration Using an Arbitrary Trihedron , 2013, Sensors.

[26]  Vincent Frémont,et al.  Extrinsic calibration between a multi-layer lidar and a camera , 2008, 2008 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems.

[27]  Andrew Zisserman,et al.  Multiple view geometry in computer visiond , 2001 .

[28]  Zhidong Deng,et al.  Extrinsic calibration of a camera and a lidar based on decoupling the rotation from the translation , 2012, 2012 IEEE Intelligent Vehicles Symposium.

[29]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[30]  David W. Scott,et al.  Multivariate Density Estimation: Theory, Practice, and Visualization , 1992, Wiley Series in Probability and Statistics.

[31]  Roland Siegwart,et al.  Extrinsic self calibration of a camera and a 3D laser range finder from natural scenes , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[32]  Silvio Savarese,et al.  Semantic structure from motion , 2011, CVPR 2011.

[33]  Saba Akram,et al.  Newton Raphson Method , 2015 .

[34]  Guy Marchal,et al.  Multimodality image registration by maximization of mutual information , 1997, IEEE Transactions on Medical Imaging.

[35]  Dimitrios G. Kottas,et al.  3D LIDAR–camera intrinsic and extrinsic calibration: Identifiability and analytical least-squares-based initialization , 2012, Int. J. Robotics Res..

[36]  Silvio Savarese,et al.  Automatic Targetless Extrinsic Calibration of a 3D Lidar and Camera by Maximizing Mutual Information , 2012, AAAI.

[37]  Gérard G. Medioni,et al.  Mutual information computation and maximization using GPU , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[38]  C. D. Kemp,et al.  Density Estimation for Statistics and Data Analysis , 1987 .

[39]  Silvio Savarese,et al.  Toward mutual information based automatic registration of 3D point clouds , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[40]  H. Cramér.  Mathematical methods of statistics , 1947 .

[41]  Robert Pless,et al.  Extrinsic Auto-calibration of a Camera and Laser Range Finder , 2003 .

[42]  Frank P. Ferrie,et al.  Automatic registration of mobile LiDAR and spherical panoramas , 2012, 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[43]  D. W. Scott,et al.  Multivariate Density Estimation, Theory, Practice and Visualization , 1992 .

[44]  Yunhui Liu,et al.  An algorithm for extrinsic parameters calibration of a camera and a laser range finder using line features , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[45]  Ryan M. Eustice,et al.  Visual localization in fused image and laser range data , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[46]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[47]  Paul Suetens,et al.  Comparative evaluation of multiresolution optimization strategies for multimodality image registration by maximization of mutual information , 1999, Medical Image Anal..

[48]  David J. Hawkes,et al.  A Strategy for Automated Multimodality Image Registration Incorporating Anatomical Knowledge and Imager Characteristics , 1993, IPMI.

[49]  Robert Pless,et al.  Extrinsic calibration of a camera and laser range finder (improves camera calibration) , 2004, 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566).

[50]  Paul Newman,et al.  Outdoor SLAM using visual appearance and laser ranging , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[51]  Mongi A. Abidi,et al.  Registration and integration of multisensor data for photorealistic scene reconstruction , 2000, Applied Imaging Pattern Recognition.

[52]  Ronald Cools,et al.  The Newton‐Raphson method , 1995 .

[53]  Gamini Dissanayake,et al.  Mutual Information based Sensor Registration and Calibration , 2006, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[54]  Cindy Cappelle,et al.  Optimal Extrinsic Calibration Between a Stereoscopic System and a LIDAR , 2013, IEEE Transactions on Instrumentation and Measurement.

[55]  Silvio Savarese,et al.  Extrinsic Calibration of a 3D Laser Scanner and an Omnidirectional Camera , 2010 .

[56]  YE Cang,et al.  Pose Estimation of an Autonomous Car by Visual Feature Correspondence and Tracking , 2013 .

[57]  Peter Cheeseman,et al.  A stochastic map for uncertain spatial relationships , 1988 .

[58]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[59]  Ryan M. Eustice,et al.  Ford Campus vision and lidar data set , 2011, Int. J. Robotics Res..

[60]  Michael Bosse,et al.  Line-based extrinsic calibration of range and image sensors , 2013, 2013 IEEE International Conference on Robotics and Automation.