A Survey on Deep Domain Adaptation for LiDAR Perception

Scalable systems for automated driving have to reliably cope with an open-world setting. This means, the perception systems are exposed to drastic domain shifts, like changes in weather conditions, time-dependent aspects, or geographic regions. Covering all domains with annotated data is impossible because of the endless variations of domains and the time-consuming and expensive annotation process. Furthermore, fast development cycles of the system additionally introduce hardware changes, such as sensor types and vehicle setups, and the required knowledge transfer from simulation.To enable scalable automated driving, it is therefore crucial to address these domain shifts in a robust and efficient manner. Over the last years, a vast amount of different domain adaptation techniques evolved. There already exists a number of survey papers for domain adaptation on camera images, however, a survey for LiDAR perception is absent. Nevertheless, LiDAR is a vital sensor for automated driving that provides detailed 3D scans of the vehicle’s surroundings. To stimulate future research, this paper presents a comprehensive review of recent progress in domain adaptation methods and formulates interesting research questions specifically targeted towards LiDAR perception.

[1]  W. Burgard,et al.  EfficientLPS: Efficient LiDAR Panoptic Segmentation , 2021, IEEE Transactions on Robotics.

[2]  Alberto L. Sangiovanni-Vincentelli,et al.  A Review of Single-Source Deep Unsupervised Visual Domain Adaptation , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[3]  Jiquan Ngiam,et al.  Pseudo-labeling for Scalable 3D Object Detection , 2021, ArXiv.

[4]  Ryo Kurazume,et al.  Learning to Drop Points for LiDAR Scan Synthesis , 2021, 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[5]  Trevor Darrell,et al.  ePointDA: An End-to-End Simulation-to-Real Domain Adaptation Framework for LiDAR Point Cloud Segmentation , 2020, AAAI.

[6]  Thomas Funkhouser,et al.  Complete & Label: A Domain Adaptation Approach to Semantic Segmentation of LiDAR Point Clouds , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Mohammed Bennamoun,et al.  Deep Learning for 3D Point Clouds: A Survey , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Ran Cheng,et al.  S3CNet: A Sparse Semantic Scene Completion Network for LiDAR Point Clouds , 2020, CoRL.

[9]  Mohamed ElHelw,et al.  Improved Semantic Segmentation of Low-Resolution 3D Point Clouds Using Supervised Domain Adaptation , 2020, 2020 2nd Novel Intelligent and Leading Emerging Sciences Conference (NILES).

[10]  C. Stachniss,et al.  Domain Transfer for Semantic Segmentation of LiDAR Data using Deep Neural Networks , 2020, 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[11]  Ana Cristina Murillo,et al.  Domain Adaptation in LiDAR Semantic Segmentation , 2020, ArXiv.

[12]  Dariu M. Gavrila,et al.  SCSSnet: Learning Spatially-Conditioned Scene Segmentation on LiDAR Point Clouds , 2020, 2020 IEEE Intelligent Vehicles Symposium (IV).

[13]  Cristiano Saltori,et al.  SF-UDA3D: Source-Free Unsupervised Domain Adaptation for LiDAR-Based 3D Object Detection , 2020, 2020 International Conference on 3D Vision (3DV).

[14]  Johann Marius Zöllner,et al.  Unsupervised Evaluation of Lidar Domain Adaptation , 2020, 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC).

[15]  Zi Huang,et al.  Progressive Graph Learning for Open-Set Domain Adaptation , 2020, ICML.

[16]  Pietro Zanuttigh,et al.  Unsupervised Domain Adaptation in Semantic Segmentation: a Review , 2020, ArXiv.

[17]  You Li,et al.  Lidar for Autonomous Driving: The Principles, Challenges, and Trends for Automotive Lidar and Perception Systems , 2020, IEEE Signal Processing Magazine.

[18]  Yohannes Kassahun,et al.  A2D2: Audi Autonomous Driving Dataset , 2020, ArXiv.

[19]  Brendan Englot,et al.  Simulation-based Lidar Super-resolution for Ground Vehicles , 2020, Robotics Auton. Syst..

[20]  Bichen Wu,et al.  SqueezeSegV3: Spatially-Adaptive Convolution for Efficient Point-Cloud Segmentation , 2020, ECCV.

[21]  Shinpei Kato,et al.  LIBRE: The Multiple 3D LiDAR Dataset , 2020, 2020 IEEE Intelligent Vehicles Symposium (IV).

[22]  Srikanth Saripalli,et al.  LiDARNet: A Boundary-Aware Domain Adaptation Model for Lidar Point Cloud Semantic Segmentation , 2020, ArXiv.

[23]  Biao Gao,et al.  SemanticPOSS: A Point Cloud Dataset with Large Quantity of Dynamic Instances , 2020, 2020 IEEE Intelligent Vehicles Symposium (IV).

[24]  Saifullahi Aminu Bello,et al.  Review: deep learning on 3D point clouds , 2020, Remote. Sens..

[25]  Dragomir Anguelov,et al.  Scalability in Perception for Autonomous Driving: Waymo Open Dataset , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  W. Stork,et al.  CNN-Based Lidar Point Cloud De-Noising in Adverse Weather , 2019, IEEE Robotics and Automation Letters.

[27]  Raoul de Charette,et al.  xMUDA: Cross-Modal Unsupervised Domain Adaptation for 3D Semantic Segmentation , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Qiang Xu,et al.  nuScenes: A Multimodal Dataset for Autonomous Driving , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Felix Heide,et al.  Seeing Through Fog Without Seeing Fog: Deep Multimodal Sensor Fusion in Unseen Adverse Weather , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Diane J. Cook,et al.  A Survey of Unsupervised Deep Domain Adaptation , 2018, ACM Trans. Intell. Syst. Technol..

[31]  Inbar Mosseri,et al.  XGAN: Unsupervised Image-to-Image Translation for many-to-many Mappings , 2017, Domain Adaptation for Visual Understanding.

[32]  Mohamed Shawky,et al.  Unsupervised Neural Sensor Models for Synthetic LiDAR Data Augmentation , 2019, ArXiv.

[33]  C.-C. Jay Kuo,et al.  PointDAN: A Multi-Scale 3D Domain Adaption Network for Point Cloud Representation , 2019, NeurIPS.

[34]  Cyrill Stachniss,et al.  RangeNet ++: Fast and Accurate LiDAR Semantic Segmentation , 2019, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[35]  Santiago Royo,et al.  An Overview of Lidar Imaging Systems for Autonomous Vehicles , 2019, Applied Sciences.

[36]  Deng Cai,et al.  Domain Adaptation for Semantic Segmentation With Maximum Squares Loss , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[37]  Guillermo Sapiro,et al.  Range Adaptation for 3D Object Detection in LiDAR , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[38]  Kaiming He,et al.  Group Normalization , 2018, International Journal of Computer Vision.

[39]  Johann Marius Zöllner,et al.  Analyzing the Cross-Sensor Portability of Neural Network Architectures for LiDAR-based Semantic Labeling , 2019, 2019 IEEE Intelligent Transportation Systems Conference (ITSC).

[40]  Dariu M. Gavrila,et al.  Cross-Sensor Deep Domain Adaptation for LiDAR Detection and Segmentation , 2019, 2019 IEEE Intelligent Vehicles Symposium (IV).

[41]  David Peter,et al.  CNN-based synthesis of realistic high-resolution LiDAR data , 2019, 2019 IEEE Intelligent Vehicles Symposium (IV).

[42]  Jiwen Lu,et al.  Spherical Fractal Convolutional Neural Networks for Point Cloud Recognition , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  Bohyung Han,et al.  Domain-Specific Batch Normalization for Unsupervised Domain Adaptation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Han Zhao,et al.  On Learning Invariant Representations for Domain Adaptation , 2019, ICML.

[45]  Julie Iskander,et al.  Domain Adaptation for Vehicle Detection from Bird's Eye View LiDAR Point Cloud Data , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[46]  Ibrahim Sobh,et al.  LiDAR Sensor modeling and Data augmentation with GANs for Autonomous driving , 2019, ArXiv.

[47]  Cyrill Stachniss,et al.  SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[48]  Jiong Yang,et al.  PointPillars: Fast Encoders for Object Detection From Point Clouds , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[49]  Joelle Pineau,et al.  Deep Generative Modeling of LiDAR Data , 2018, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[50]  Patrick Pérez,et al.  ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[51]  Kurt Keutzer,et al.  SqueezeSegV2: Improved Model Structure and Unsupervised Domain Adaptation for Road-Object Segmentation from a LiDAR Point Cloud , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[52]  Leonidas J. Guibas,et al.  FlowNet3D: Learning Scene Flow in 3D Point Clouds , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[53]  Yue Wang,et al.  Dynamic Graph CNN for Learning on Point Clouds , 2018, ACM Trans. Graph..

[54]  Ryo Kurazume,et al.  Fukuoka datasets for place categorization , 2019, Int. J. Robotics Res..

[55]  Hyo-Eun Kim,et al.  Batch-Instance Normalization for Adaptively Style-Invariant Neural Networks , 2018, NeurIPS.

[56]  Tatsuya Harada,et al.  Open Set Domain Adaptation by Backpropagation , 2018, ECCV.

[57]  Ali Farhadi,et al.  YOLOv3: An Incremental Improvement , 2018, ArXiv.

[58]  Alberto L. Sangiovanni-Vincentelli,et al.  A LiDAR Point Cloud Generator: from a Virtual World to Autonomous Driving , 2018, ICMR.

[59]  Ming-Hsuan Yang,et al.  Learning to Adapt Structured Output Space for Semantic Segmentation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[60]  Subhransu Maji,et al.  SPLATNet: Sparse Lattice Networks for Point Cloud Processing , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[61]  François Goulette,et al.  Paris-Lille-3D: A large and high-quality ground-truth urban point cloud dataset for automatic segmentation and classification , 2017, Int. J. Robotics Res..

[62]  Jung-Woo Ha,et al.  StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[63]  Yin Zhou,et al.  VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[64]  Taesung Park,et al.  CyCADA: Cycle-Consistent Adversarial Domain Adaptation , 2017, ICML.

[65]  Vittorio Murino,et al.  Minimal-Entropy Correlation Alignment for Unsupervised Deep Domain Adaptation , 2017, ICLR.

[66]  Germán Ros,et al.  CARLA: An Open Urban Driving Simulator , 2017, CoRL.

[67]  Juergen Gall,et al.  Open Set Domain Adaptation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[68]  Lior Wolf,et al.  One-Sided Unsupervised Domain Mapping , 2017, NIPS.

[69]  Min Sun,et al.  No More Discrimination: Cross City Adaptation of Road Scene Segmenters , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[70]  Victor S. Lempitsky,et al.  Escape from Cells: Deep Kd-Networks for the Recognition of 3D Point Cloud Models , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[71]  Subhransu Maji,et al.  3D Shape Segmentation with Projective Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[72]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[73]  Lior Wolf,et al.  Unsupervised Cross-Domain Image Generation , 2016, ICLR.

[74]  Jiaying Liu,et al.  Revisiting Batch Normalization For Practical Domain Adaptation , 2016, ICLR.

[75]  Andrea Vedaldi,et al.  Instance Normalization: The Missing Ingredient for Fast Stylization , 2016, ArXiv.

[76]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[77]  Simon Osindero,et al.  Conditional Generative Adversarial Nets , 2014, ArXiv.

[78]  Andreas Geiger,et al.  Are we ready for autonomous driving? The KITTI vision benchmark suite , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[79]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.