MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Driving

Nowadays, autonomous cars can drive smoothly in ordinary cases, and it is widely recognized that realistic sensor simulation will play a critical role in solving remaining corner cases by simulating them. To this end, we propose an autonomous driving simulator based upon neural radiance fields (NeRFs). Compared with existing works, ours has three notable features: (1) Instance-aware. Our simulator models the foreground instances and background environments separately with independent networks so that the static (e.g., size and appearance) and dynamic (e.g., trajectory) properties of instances can be controlled separately. (2) Modular. Our simulator allows flexible switching between different modern NeRF-related backbones, sampling strategies, input modalities, etc. We expect this modular design to boost academic progress and industrial deployment of NeRF-based autonomous driving simulation. (3) Realistic. Our simulator set new state-of-the-art photo-realism results given the best module selection. Our simulator will be open-sourced while most of our counterparts are not. Project page: https://open-air-sun.github.io/mars/.

[1]  R. Urtasun,et al.  UniSim: A Neural Closed-Loop Sensor Simulator , 2023, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Pengfei Li,et al.  Unsupervised Road Anomaly Detection with Language Anchors , 2023, 2023 IEEE International Conference on Robotics and Automation (ICRA).

[3]  C. Theobalt,et al.  F2-NeRF: Fast Neural Radiance Field Training with Free Camera Trajectories , 2023, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Jason Y. Zhang,et al.  SUDS: Scalable Urban Dynamic Scenes , 2023, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Junge Zhang,et al.  S-NeRF: Neural Radiance Fields for Street Views , 2023, ICLR.

[6]  Pengfei Li,et al.  LODE: Locally Conditioned Eikonal Implicit Scene Completion from Sparse LiDAR , 2023, 2023 IEEE International Conference on Robotics and Automation (ICRA).

[7]  Angjoo Kanazawa,et al.  Nerfstudio: A Modular Framework for Neural Radiance Field Development , 2023, SIGGRAPH.

[8]  Qichao Zhang,et al.  STEPS: Joint Self-supervised Nighttime Image Enhancement and Depth Estimation , 2023, 2023 IEEE International Conference on Robotics and Automation (ICRA).

[9]  T. Zhang,et al.  ADAPT: Action-aware Driving Caption Transformer , 2023, 2023 IEEE International Conference on Robotics and Automation (ICRA).

[10]  B. Recht,et al.  K-Planes: Explicit Radiance Fields in Space, Time, and Appearance , 2023, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Jifeng Dai,et al.  Planning-oriented Autonomous Driving , 2022, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Beiwen Tian,et al.  VIBUS: Data-efficient 3D Scene Parsing with VIewpoint Bottleneck and Uncertainty-Spectrum Modeling , 2022, ISPRS Journal of Photogrammetry and Remote Sensing.

[13]  Andreas Geiger,et al.  MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction , 2022, NeurIPS.

[14]  T. Funkhouser,et al.  Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Zaiqing Nie,et al.  DAIR-V2X: A Large-Scale Dataset for Vehicle-Infrastructure Cooperative 3D Object Detection , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Andreas Geiger,et al.  Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation , 2022, 2022 International Conference on 3D Vision (3DV).

[17]  Andreas Geiger,et al.  TensoRF: Tensorial Radiance Fields , 2022, ECCV.

[18]  T. Müller,et al.  Instant neural graphics primitives with a multiresolution hash encoding , 2022, ACM Trans. Graph..

[19]  Jonathan T. Barron,et al.  Urban Radiance Fields , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Pratul P. Srinivasan,et al.  Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Hao Zhao,et al.  PQ-Transformer: Jointly Parsing 3D Objects and Layouts from Point Clouds , 2021, IEEE Robotics and Automation Letters.

[22]  D. Ramanan,et al.  Depth-supervised NeRF: Fewer Views and Faster Training for Free , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Pratul P. Srinivasan,et al.  Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[24]  R. Urtasun,et al.  GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Driving , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Andreas Geiger,et al.  GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Felix Heide,et al.  Neural Scene Graphs for Dynamic Scenes , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Pratul P. Srinivasan,et al.  NeRF , 2020, ECCV.

[28]  Naila Murray,et al.  Virtual KITTI 2 , 2020, ArXiv.

[29]  D. Manocha,et al.  AADS: Augmented autonomous driving simulation using data-driven algorithms , 2019, Science Robotics.

[30]  Alexei A. Efros,et al.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[31]  V. Koltun,et al.  CARLA: An Open Urban Driving Simulator , 2017, CoRL.

[32]  Andreas Geiger,et al.  Vision meets robotics: The KITTI dataset , 2013, Int. J. Robotics Res..

[33]  Andreas Geiger,et al.  Are we ready for autonomous driving? The KITTI vision benchmark suite , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.