CADSim: Robust and Scalable in-the-wild 3D Reconstruction for Controllable Sensor Simulation

: In this supplementary material, we provide additional details on our method and experiments, and then show additional application results using CADSim. For our method, we describe how we create a shared low-dimensional representation space for optimizing over a set of CAD models (Sec A.1), provide details on our selected appearance representation (Sec A.2), and the exact inference optimization procedure performed and hyperparameters used CADSim (Sec A.3). For our experiments, we first provide implementation details of the baselines compared against (Sec B), as well as dataset and metric details (Sec C). We then report thorough ablations on our choice of geometry and appearance (Sec D.1), demonstrate the robustness of CADSim to data noise (Sec D.2), show our approach applied to non-vehicle objects (Sec D.3), and show more experiment results of our model improving perception evaluation (Sec D.5). Finally, we show additional applications of CADSim. We show using CADSim for multi-sensor simulation examples for scenario replay (Sec E.1) and mixed reality (Sec E.2), as well as showing CADSim naturally supporting texture transfer for creating diverse assets (Sec E.3). Additionally, we include a supplementary video, supplementary 56.mp4 providing an overview of our methodology, as well as video results on novel-view synthesis, and realistic multi-sensor simulation.

[1]  Junyan Zhu,et al.  On Aliased Resizing and Surprising Subtleties in GAN Evaluation , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Yuanzhen Li,et al.  SAMURAI: Shape And Material from Unconstrained Real-world Arbitrary Image collections , 2022, NeurIPS.

[3]  Alexei A. Efros,et al.  Share With Thy Neighbors: Single-View Reconstruction by Cross-Instance Consistency , 2022, ECCV.

[4]  S. Fidler,et al.  AUV-Net: Learning Aligned UV Maps for Texture Transfer and Synthesis , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  T. Müller,et al.  Instant neural graphics primitives with a multiresolution hash encoding , 2022, ACM Trans. Graph..

[6]  Benjamin Recht,et al.  Plenoxels: Radiance Fields without Neural Networks , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Jonathan T. Barron,et al.  RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  A. Jacobson,et al.  Large steps in inverse rendering of geometry , 2021, ACM Trans. Graph..

[9]  S. Fidler,et al.  Extracting Triangular 3D Models, Materials, and Lighting From Images , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Federico Tombari,et al.  Neural Fields in Visual Computing and Beyond , 2021, Comput. Graph. Forum.

[11]  Sanja Fidler,et al.  Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis , 2021, NeurIPS.

[12]  Sanja Fidler,et al.  DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer , 2021, NeurIPS.

[13]  Deva Ramanan,et al.  NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild , 2021, NeurIPS.

[14]  Kun Jiang,et al.  PandaSet: Advanced Sensor Suite Dataset for Autonomous Driving , 2021, 2021 IEEE International Intelligent Transportation Systems Conference (ITSC).

[15]  J.-Y. Zhu,et al.  Advances in Neural Rendering , 2021, SIGGRAPH Courses.

[16]  D. Ramanan,et al.  Depth-supervised NeRF: Fewer Views and Faster Training for Free , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  C. Theobalt,et al.  NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction , 2021, NeurIPS.

[18]  Paul Debevec,et al.  NeRFactor , 2021, ACM Trans. Graph..

[19]  Noah Snavely,et al.  KeypointDeformer: Unsupervised 3D Keypoint Discovery for Shape Control , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Andreas Geiger,et al.  UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[21]  Gerard Pons-Moll,et al.  Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenes , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Noah Snavely,et al.  PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Hao Su,et al.  MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[24]  Jonathan T. Barron,et al.  Baking Neural Radiance Fields for Real-Time View Synthesis , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[25]  Ren Ng,et al.  PlenOctrees for Real-time Rendering of Neural Radiance Fields , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[26]  Vladlen Koltun,et al.  Large Batch Simulation for Deep Reinforcement Learning , 2021, ICLR.

[27]  Charles T. Loop,et al.  Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Vladimir G. Kim,et al.  Joint Learning of 3D Shape Retrieval and Deformation , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Raquel Urtasun,et al.  Mending Neural Implicit Modeling for 3D Vehicle Reconstruction in the Wild , 2021, 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV).

[30]  R. Urtasun,et al.  GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Driving , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  R. Urtasun,et al.  AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehicles , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Raquel Urtasun,et al.  SceneGen: Learning to Generate Realistic Traffic Scenes , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Jiajun Wu,et al.  Object-Centric Neural Scene Rendering , 2020, ArXiv.

[34]  Pratul P. Srinivasan,et al.  Learned Initializations for Optimizing Coordinate-Based Neural Representations , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Raquel Urtasun,et al.  Recovering and Simulating Pedestrians in the Wild , 2020, CoRL.

[36]  Jaakko Lehtinen,et al.  Modular primitives for high-performance differentiable rendering , 2020, ACM Trans. Graph..

[37]  Simon Lucey,et al.  SDF-SRN: Learning Signed Distance 3D Object Reconstruction from Static Images , 2020, NeurIPS.

[38]  Kai Zhang,et al.  NeRF++: Analyzing and Improving Neural Radiance Fields , 2020, ArXiv.

[39]  Alex Trevithick,et al.  GRF: Learning a General Radiance Field for 3D Representation and Rendering , 2020, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[40]  Nan Yang,et al.  Learning Monocular 3D Vehicle Detection Without 3D Bounding Box Labels , 2020, GCPR.

[41]  Zihao Wang,et al.  Weakly-supervised 3D Shape Completion in the Wild , 2020, ECCV.

[42]  Jitendra Malik,et al.  Shape and Viewpoint without Keypoints , 2020, ECCV.

[43]  Abhinav Gupta,et al.  Implicit Mesh Reconstruction from Unannotated Image Collections , 2020, ArXiv.

[44]  Gordon Wetzstein,et al.  Implicit Neural Representations with Periodic Activation Functions , 2020, NeurIPS.

[45]  Raquel Urtasun,et al.  LiDARsim: Realistic LiDAR Simulation by Leveraging the Real World , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  Yichao Zhou,et al.  ManifoldPlus: A Robust and Scalable Watertight Manifold Surface Generation Method for Triangle Soups , 2020, ArXiv.

[47]  Dumitru Erhan,et al.  SurfelGAN: Synthesizing Realistic Sensor Data for Autonomous Driving , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  Jingwei Huang,et al.  Deformation-Aware 3D Model Embedding and Retrieval , 2020, ECCV.

[49]  Raquel Urtasun,et al.  Physically Realizable Adversarial Examples for LiDAR Object Detection , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[50]  Richard A. Newcombe,et al.  Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstruction , 2020, ECCV.

[51]  Ronen Basri,et al.  Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance , 2020, NeurIPS.

[52]  Pratul P. Srinivasan,et al.  NeRF , 2020, ECCV.

[53]  Ross B. Girshick,et al.  PointRend: Image Segmentation As Rendering , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[54]  Vladimir G. Kim,et al.  Neural Cages for Detail-Preserving 3D Deformations , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[55]  Wan-Yen Lo,et al.  Accelerating 3D deep learning with PyTorch3D , 2019, SIGGRAPH Asia 2020 Courses.

[56]  Alec Jacobson,et al.  Cubic stylization , 2019, ACM Trans. Graph..

[57]  S. Fidler,et al.  Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer , 2019, NeurIPS.

[58]  Duygu Ceylan,et al.  DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction , 2019, NeurIPS.

[59]  Thomas Brox,et al.  What Do Single-View 3D Reconstruction Networks Learn? , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[60]  D. Cremers,et al.  DirectShape: Direct Photometric Alignment of Shape Priors for Visual Vehicle Pose and Shape Estimation , 2019, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[61]  Dimitrios Tzionas,et al.  Expressive Body Capture: 3D Hands, Face, and Body From a Single Image , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[62]  Hao Li,et al.  Soft Rasterizer: A Differentiable Renderer for Image-Based 3D Reasoning , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[63]  Richard A. Newcombe,et al.  DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[64]  Yi Zhou,et al.  On the Continuity of Rotation Representations in Neural Networks , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[65]  Sebastian Nowozin,et al.  Occupancy Networks: Learning 3D Reconstruction in Function Space , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[66]  Alec Jacobson,et al.  Paparazzi , 2018, ACM Trans. Graph..

[67]  Alexey Dosovitskiy,et al.  Unsupervised Learning of Shape and Pose with Differentiable Point Clouds , 2018, NeurIPS.

[68]  Bolei Zhou,et al.  Single Image Intrinsic Decomposition Without a Single Intrinsic Image , 2018, ECCV.

[69]  Jiajun Wu,et al.  Learning Shape Priors for Single-View 3D Completion and Reconstruction , 2018, ECCV.

[70]  Martial Hebert,et al.  PCN: Point Completion Network , 2018, 2018 International Conference on 3D Vision (3DV).

[71]  Noah Snavely,et al.  Layer-structured 3D Scene Inference via View Synthesis , 2018, ECCV.

[72]  Jitendra Malik,et al.  Learning Category-Specific Mesh Reconstruction from Image Collections , 2018, ECCV.

[73]  Leonidas J. Guibas,et al.  Robust Watertight Manifold Surface Generation Method for ShapeNet Models , 2018, ArXiv.

[74]  Tatsuya Harada,et al.  Neural 3D Mesh Renderer , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[75]  Jiajun Wu,et al.  MarrNet: 3D Shape Reconstruction via 2.5D Sketches , 2017, NIPS.

[76]  V. Koltun,et al.  CARLA: An Open Urban Driving Simulator , 2017, CoRL.

[77]  Christopher Kulla,et al.  Physically based shading in theory and practice , 2014, SIGGRAPH '14.

[78]  Sepp Hochreiter,et al.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium , 2017, NIPS.

[79]  S. Lucey,et al.  Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction , 2017, AAAI.

[80]  Thierry Chateau,et al.  Deep MANTA: A Coarse-to-Fine Many-Task Network for Joint 2D and 3D Vehicle Analysis from Monocular Image , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[81]  Ross B. Girshick,et al.  Mask R-CNN , 2017, 1703.06870.

[82]  Jörg Stückler,et al.  SAMP: Shape and Motion Priors for 4D Vehicle Reconstruction , 2017, 2017 IEEE Winter Conference on Applications of Computer Vision (WACV).

[83]  Michael J. Black,et al.  3D Menagerie: Modeling the 3D Shape and Pose of Animals , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[84]  Jiajun Wu,et al.  Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling , 2016, NIPS.

[85]  Andrew Sanders,et al.  An Introduction to Unreal Engine 4 , 2016 .

[86]  Jörg Stückler,et al.  Joint Object Pose Estimation and Shape Reconstruction in Urban Street Scenes Using 3D Shape Priors , 2016, GCPR.

[87]  Michael J. Black,et al.  SMPL: A Skinned Multi-Person Linear Model , 2023 .

[88]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[89]  Michael J. Black,et al.  OpenDR: An Approximate Differentiable Renderer , 2014, ECCV.

[90]  Brian E. Smits,et al.  Practical physically-based shading in film and game production , 2012, SIGGRAPH '12.

[91]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[92]  K. Torrance,et al.  Microfacet Models for Refraction through Rough Surfaces , 2007, Rendering Techniques.

[93]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[94]  Matthias Zwicker,et al.  Surfels: surface elements as rendering primitives , 2000, SIGGRAPH.

[95]  Thomas Ertl,et al.  Computer Graphics - Principles and Practice, 3rd Edition , 2014 .

[96]  William E. Lorensen,et al.  Marching cubes: A high resolution 3D surface construction algorithm , 1987, SIGGRAPH.

[97]  Bui Tuong Phong Illumination for computer generated pictures , 1975, Commun. ACM.

[98]  Leonidas J. Guibas,et al.  ShapeFlow: Learnable Deformation Flows Among 3D Shapes , 2020, NeurIPS.

[99]  Brian Karis,et al.  Real Shading in Unreal Engine 4 by , 2013 .

[100]  W. Marsden I and J , 2012 .

[101]  Frederick R. Forst,et al.  On robust estimation of the location parameter , 1980 .

[102]  E. Land,et al.  Lightness and retinex theory. , 1971, Journal of the Optical Society of America.