Adversarial Training on Point Clouds for Sim-to-Real 3D Object Detection

In this work we address the problem of 3D object detection from point clouds in data-limited environments. Training with simulated data is a common approach in such scenarios; however a sim-to-real gap exists between clean and crisp simulated clouds and noisy real clouds. Previous sim-to-real approaches for processing point cloud scenes have compressed clouds into 2D and used 2D transfer techniques. However, this may compress useful 3D information and does not effectively reason about the unstructured nature of point cloud data. We thus propose a 3D adversarial training architecture that leverages an adaptive sampling module to reason about the unstructured nature of point cloud data. Our approach encourages the 3D feature encoder to extract features that are invariant across simulated and real scenes. We validate our approach in the context of the DARPA Subterranean Challenge and demonstrate that our 3D adversarial training architecture improves 3D object detection performance by up to 15% depending on the data representation.