Lasers to Events: Automatic Extrinsic Calibration of Lidars and Event Cameras

Despite significant academic and corporate efforts, autonomous driving under adverse visual conditions still proves challenging. As neuromorphic technology has ma-tured, its application to robotics and autonomous vehicle systems has become an area of active research. Low-light and latency-demanding situations can benefit. To enable event cameras to operate alongside staple sensors like lidar in perception tasks, we propose a direct, temporally-decoupled calibration method between event cameras and lidars. The high dynamic range and low-light operation of event cameras are exploited to directly register lidar laser returns, allowing information-based correlation methods to optimize for the 6-DoF extrinsic calibration between the two sensors. This paper presents the first direct calibration method between event cameras and lidars, removing de-pendencies on frame-based camera intermediaries and/or highly-accurate hand measurements. Code will be made publicly available.

[1]  Chiara Bartolozzi,et al.  Event-Based Vision: A Survey , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Davide Scaramuzza,et al.  ESL: Event-based Structured Light , 2021, 2021 International Conference on 3D Vision (3DV).

[3]  Davide Scaramuzza,et al.  How to Calibrate Your Event Camera , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[4]  Julian F. P. Kooij,et al.  A Joint Extrinsic Calibration Tool for Radar, Camera and Lidar , 2021, IEEE Transactions on Intelligent Vehicles.

[5]  Davide Scaramuzza,et al.  DSEC: A Stereo Event Camera Dataset for Driving Scenarios , 2021, IEEE Robotics and Automation Letters.

[6]  Joseph Walsh,et al.  Sensor and Sensor Fusion Technology in Autonomous Vehicles: A Review , 2021, Sensors.

[7]  Vladlen Koltun,et al.  High Speed and High Dynamic Range Video with an Event Camera , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Daniel Matolin,et al.  5.10 A 1280×720 Back-Illuminated Stacked Temporal Contrast Event-Based Vision Sensor with 4.86µm Pixels, 1.066GEPS Readout, Programmable Event-Rate Controller and Compressive Data-Formatting Pipeline , 2020, 2020 IEEE International Solid- State Circuits Conference - (ISSCC).

[9]  Jun Zhang,et al.  Geometric calibration for LiDAR-camera system fusing 3D-2D and 3D-3D point correspondences. , 2020, Optics express.

[10]  Sufyan Almajali,et al.  A Systematic Review on Fusion Techniques and Approaches Used in Applications , 2020, IEEE Access.

[11]  P. Newman,et al.  The Oxford Radar RobotCar Dataset: A Radar Extension to the Oxford RobotCar Dataset , 2019, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[12]  Nakju Lett Doh,et al.  Automatic targetless camera–LIDAR calibration by aligning edge with Gaussian mixture model , 2019, J. Field Robotics.

[13]  Joel Nothman,et al.  SciPy 1.0-Fundamental Algorithms for Scientific Computing in Python , 2019, ArXiv.

[14]  Vladlen Koltun,et al.  Events-To-Video: Bringing Modern Computer Vision to Event Cameras , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Lipu Zhou,et al.  Automatic Extrinsic Calibration of a Camera and a 3D LiDAR Using Line and Plane Correspondences , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[16]  David San Segundo Bello,et al.  Front and Back Illuminated Dynamic and Active Pixel Vision Sensors Comparison , 2018, IEEE Transactions on Circuits and Systems II: Express Briefs.

[17]  Vijay Kumar,et al.  The Multivehicle Stereo Event Camera Dataset: An Event Camera Dataset for 3D Perception , 2018, IEEE Robotics and Automation Letters.

[18]  Paul Newman,et al.  1 year, 1000 km: The Oxford RobotCar dataset , 2017, Int. J. Robotics Res..

[19]  Bhaskar Choubey,et al.  Advances on CMOS image sensors , 2016 .

[20]  Silvio Savarese,et al.  Automatic Extrinsic Calibration of Vision and Lidar by Maximizing Mutual Information , 2015, J. Field Robotics.

[21]  Roland Siegwart,et al.  Adaptive pulsed laser line extraction for terrain reconstruction using a dynamic vision sensor , 2014, Front. Neurosci..

[22]  Andreas Geiger,et al.  Vision meets robotics: The KITTI dataset , 2013, Int. J. Robotics Res..

[23]  Roland Siegwart,et al.  Rolling Shutter Camera Calibration , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Andreas Geiger,et al.  Automatic camera and range sensor calibration using a single shot , 2012, 2012 IEEE International Conference on Robotics and Automation.

[25]  P. Fereyre,et al.  BACK ILLUMINATED SYSTEM-ON-CHIP FOR NIGHT VISION , 2012 .

[26]  Juan I. Nieto,et al.  A mutual information approach to automatic calibration of camera and lidar in natural environments , 2012 .

[27]  Heiko Hirschmüller,et al.  Stereo Processing by Semiglobal Matching and Mutual Information , 2008, IEEE Trans. Pattern Anal. Mach. Intell..

[28]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[29]  Gérard G. Medioni,et al.  Object modeling by registration of multiple range images , 1991, Proceedings. 1991 IEEE International Conference on Robotics and Automation.

[30]  Khosrow Dehnad,et al.  Density Estimation for Statistics and Data Analysis , 1987 .

[31]  P. J. Green,et al.  Density Estimation for Statistics and Data Analysis , 1987 .