End-to-End Optimization of LiDAR Beam Configuration for 3D Object Detection and Localization

Existing learning methods for LiDAR-based applications use 3D points scanned under a pre-determined beam configuration, e.g., the elevation angles of beams are often evenly distributed. Those fixed configurations are task-agnostic, so simply using them can lead to sub-optimal performance. In this work, we take a new route to learn to optimize the LiDAR beam configuration for a given application. Specifically, we propose a reinforcement learning-based learning-to-optimize (RL-L2O) framework to automatically optimize the beam configuration in an end-to-end manner for different LiDAR-based applications. The optimization is guided by the final performance of the target task and thus our method can be integrated easily with any LiDAR-based application as a simple drop-in module. The method is especially useful when a low-resolution (low-cost) LiDAR is needed, for instance, for system deployment at a massive scale. We use our method to search for the beam configuration of a low-resolution LiDAR for two important tasks: 3D object detection and localization. Experiments show that the proposed RL-L2O method improves the performance in both tasks significantly compared to the baseline methods. We believe that a combination of our method with the recent advances of programmable LiDARs can start a new research direction for LiDAR-based active perception. The code is publicly available at github.com/vniclas/lidar_beam_selection.

[1]  Paul Newman,et al.  The Oxford Radar RobotCar Dataset: A Radar Extension to the Oxford RobotCar Dataset , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[2]  ZhangJi,et al.  Low-drift and real-time lidar odometry and mapping , 2017 .

[3]  Yan Wang,et al.  Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Driving , 2019, ICLR.

[4]  Lu Feng,et al.  A robust pose graph approach for city scale LiDAR mapping , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[5]  Sebastian Ramos,et al.  The Cityscapes Dataset for Semantic Urban Scene Understanding , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Luc Van Gool,et al.  Weakly Supervised 3D Object Detection from Lidar Point Cloud , 2020, ECCV.

[7]  Bo Li,et al.  SECOND: Sparsely Embedded Convolutional Detection , 2018, Sensors.

[8]  Ayan Chakrabarti,et al.  Towards a MEMS-based Adaptive LIDAR , 2020, 2020 International Conference on 3D Vision (3DV).

[9]  Yong Liu,et al.  Parse geometry from a line: Monocular depth estimation with partial laser observation , 2016, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[10]  Torsten Sattler,et al.  Benchmarking 6DOF Outdoor Visual Localization in Changing Conditions , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[11]  Hang Lei,et al.  Hyperparameter Optimization for Machine Learning Models Based on Bayesian Optimization , 2019 .

[12]  Simon Lacroix,et al.  ICP-based pose-graph SLAM , 2016, 2016 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR).

[13]  Anath Fischer,et al.  3D Point Cloud Registration for Localization Using a Deep Neural Network Auto-Encoder , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Ling Shao,et al.  Dynamical Hyperparameter Optimization via Deep Reinforcement Learning in Tracking , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Kenji Koide,et al.  Automatic Hyper-Parameter Tuning for Black-box LiDAR Odometry , 2021, 2021 IEEE International Conference on Robotics and Automation (ICRA).

[16]  Yingtao Ding,et al.  A Compact Omnidirectional Laser Scanner Based on an Electrothermal Tripod Mems Mirror for Lidar Please Leave , 2019, 2019 20th International Conference on Solid-State Sensors, Actuators and Microsystems & Eurosensors XXXIII (TRANSDUCERS & EUROSENSORS XXXIII).

[17]  Xiaogang Wang,et al.  From Points to Parts: 3D Object Detection From Point Cloud With Part-Aware and Part-Aggregation Network , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Luc Van Gool,et al.  Improving Point Cloud Semantic Segmentation by Learning 3D Object Detection , 2020, 2021 IEEE Winter Conference on Applications of Computer Vision (WACV).

[19]  Kenji Narumi,et al.  Liquid crystal-tunable optical phased array for LiDAR applications , 2021, OPTO.

[20]  Gérard G. Medioni,et al.  Object modelling by registration of multiple range images , 1992, Image Vis. Comput..

[21]  Francesc Moreno-Noguer,et al.  Low Resolution Lidar-Based Multi-Object Tracking for Driving Applications , 2017, ROBOT.

[22]  Yasuyuki Matsushita,et al.  Efficient Large-Scale Point Cloud Registration Using Loop Closures , 2015, 2015 International Conference on 3D Vision.

[23]  Gordon Wetzstein,et al.  Deep Adaptive LiDAR: End-to-end Optimization of Sampling and Depth Completion at Low Sampling Rates , 2020, 2020 IEEE International Conference on Computational Photography (ICCP).

[24]  Jia Wu,et al.  Efficient hyperparameter optimization through model-based reinforcement learning , 2020, Neurocomputing.

[25]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[26]  Andreas Geiger,et al.  Vision meets robotics: The KITTI dataset , 2013, Int. J. Robotics Res..

[27]  Peter Stone,et al.  Reinforcement learning , 2019, Scholarpedia.

[28]  Wengang Zhou,et al.  Voxel R-CNN: Towards High Performance Voxel-based 3D Object Detection , 2020, AAAI.

[29]  Xiaogang Wang,et al.  PointRCNN: 3D Object Proposal Generation and Detection From Point Cloud , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Yiming Zhao,et al.  3D Vehicle Detection Using Camera and Low-Resolution LiDAR , 2021, ArXiv.

[31]  Jiong Yang,et al.  PointPillars: Fast Encoders for Object Detection From Point Clouds , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  François Chollet,et al.  Xception: Deep Learning with Depthwise Separable Convolutions , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Jiarong Lin,et al.  Loam livox: A fast, robust, high-precision LiDAR odometry and mapping package for LiDARs of small FoV , 2019, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[34]  Yan Wang,et al.  Pseudo-LiDAR From Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Guy Gilboa,et al.  Adaptive LiDAR Sampling and Depth Completion Using Ensemble Variance , 2020, IEEE Transactions on Image Processing.

[36]  Ji Zhang,et al.  Low-drift and real-time lidar odometry and mapping , 2017, Auton. Robots.

[37]  Paul Newman,et al.  Generating All the Roads to Rome: Road Layout Randomization for Improved Road Marking Segmentation , 2019, 2019 IEEE Intelligent Transportation Systems Conference (ITSC).