Dynamic Semantic Occupancy Mapping using 3D Scene Flow and Closed-Form Bayesian Inference

This paper reports on a dynamic semantic mapping framework that incorporates 3D scene flow measurements into a closed-form Bayesian inference model. Existence of dynamic objects in the environment cause artifacts and traces in current mapping algorithms, leading to an inconsistent map posterior. We leverage state-of-the-art semantic segmentation and 3D flow estimation using deep learning to provide measurements for map inference. We develop a continuous (i.e., can be queried at arbitrary resolution) Bayesian model that propagates the scene with flow and infers a 3D semantic occupancy map with better performance than its static counterpart. Experimental results using publicly available data sets show that the proposed framework generalizes its predecessors and improves over direct measurements from deep neural networks consistently.

[1]  Zhuwen Li,et al.  PointPWC-Net: A Coarse-to-Fine Network for Supervised and Self-Supervised Scene Flow Estimation on 3D Point Clouds , 2019, ArXiv.

[2]  Margrit Gelautz,et al.  SLAMANTIC - Leveraging Semantics to Improve VSLAM in Dynamic Environments , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[3]  Shichao Yang,et al.  Semantic 3D occupancy mapping through efficient high order CRFs , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[4]  Leonidas J. Guibas,et al.  FlowNet3D: Learning Scene Flow in 3D Point Clouds , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Dieter Fox,et al.  DA-RNN: Semantic Mapping with Data Associated Recurrent Neural Networks , 2017, Robotics: Science and Systems.

[6]  Alexandre Boulch,et al.  FLOT: Scene Flow on Point Clouds Guided by Optimal Transport , 2020, ECCV.

[7]  Luis Miguel Bergasa,et al.  On combining visual SLAM and dense scene flow to increase the robustness of localization and mapping in dynamic environments , 2012, 2012 IEEE International Conference on Robotics and Automation.

[8]  Jun Zhang,et al.  Dynamic SLAM: The Need For Speed , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[9]  Avideh Zakhor,et al.  Temporal LiDAR Frame Prediction for Autonomous Driving , 2020, 2020 International Conference on 3D Vision (3DV).

[10]  Ryan M. Eustice,et al.  Bayesian Spatial Kernel Smoothing for Scalable Dense Semantic Mapping , 2020, IEEE Robotics and Automation Letters.

[11]  Xin Zhang,et al.  End to End Learning for Self-Driving Cars , 2016, ArXiv.

[12]  Kris M. Kitani,et al.  Unsupervised Sequence Forecasting of 100,000 Points for Unsupervised Trajectory Forecasting , 2020, ArXiv.

[13]  Thomas Brox,et al.  A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Paul H. J. Kelly,et al.  SLAM++: Simultaneous Localisation and Mapping at the Level of Objects , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Shi-Min Hu,et al.  ClusterSLAM: A SLAM backend for simultaneous rigid body clustering and motion estimation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[16]  Lourdes Agapito,et al.  MaskFusion: Real-Time Recognition, Tracking and Reconstruction of Multiple Moving Objects , 2018, 2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[17]  Cyrill Stachniss,et al.  SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[18]  Jaime Valls Miró,et al.  Warped Gaussian Processes Occupancy Mapping With Uncertain Inputs , 2017, IEEE Robotics and Automation Letters.

[19]  Jörg Stückler,et al.  EM-Fusion: Dynamic Object-Level SLAM With Probabilistic Data Association , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[20]  Michael Milford,et al.  Semantics for Robotic Mapping, Perception and Interaction: A Survey , 2021, Found. Trends Robotics.

[21]  Gordon Wyeth,et al.  Place categorization and semantic mapping on a mobile robot , 2015, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[22]  Brendan Englot,et al.  Learning-Aided 3-D Occupancy Mapping With Bayesian Generalized Kernel Inference , 2019, IEEE Transactions on Robotics.

[23]  Leonidas J. Guibas,et al.  PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space , 2017, NIPS.

[24]  Michael Milford,et al.  Meaningful maps with object-oriented semantic mapping , 2016, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[25]  Jun Zhang,et al.  A Hierarchical Framework for Collaborative Probabilistic Semantic Mapping , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[26]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Stefan Leutenegger,et al.  SemanticFusion: Dense 3D semantic mapping with convolutional neural networks , 2016, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[28]  Li Sun,et al.  Recurrent-OctoMap: Learning State-Based Map Refinement for Long-Term Semantic Mapping With 3-D-Lidar Data , 2018, IEEE Robotics and Automation Letters.

[29]  Yue Bai,et al.  DSOD: DSO in Dynamic Environments , 2019, IEEE Access.

[30]  Brendan Englot,et al.  Fast, accurate gaussian process occupancy maps via test-data octrees and nested Bayesian fusion , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[31]  Bernard Ghanem,et al.  SCTN: Sparse Convolution-Transformer Network for Scene Flow Estimation , 2021, ArXiv.

[32]  Gabriel Oliver,et al.  Sparse Gaussian process for online seagrass semantic mapping , 2021, Expert Syst. Appl..

[33]  Luca Carlone,et al.  3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans , 2020, RSS 2020.

[34]  Odest Chadwicke Jenkins,et al.  Semantic Mapping with Simultaneous Object Detection and Localization , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[35]  Jiaqi Gu,et al.  SDP-Net: Scene Flow Based Real-Time Object Detection and Prediction from Sequential 3D Point Clouds , 2020, ACCV.

[36]  Stefan Leutenegger,et al.  Fusion++: Volumetric Object-Level SLAM , 2018, 2018 International Conference on 3D Vision (3DV).

[37]  Andreas Geiger,et al.  Object scene flow for autonomous vehicles , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Nikolas Brasch,et al.  Semantic Monocular SLAM for Highly Dynamic Environments , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[39]  Javier Civera,et al.  DynaSLAM: Tracking, Mapping, and Inpainting in Dynamic Scenes , 2018, IEEE Robotics and Automation Letters.

[40]  Marc Pollefeys,et al.  Learning Priors for Semantic 3D Reconstruction , 2018, ECCV.

[41]  Aleksandra Faust,et al.  Learning Navigation Behaviors End-to-End With AutoRL , 2018, IEEE Robotics and Automation Letters.

[42]  Marc Pollefeys,et al.  Self-Supervised Learning of Non-Rigid Residual Flow and Ego-Motion , 2020, 2020 International Conference on 3D Vision (3DV).

[43]  Jie Li,et al.  Gaussian Processes Semantic Map Representation , 2017, ArXiv.

[44]  Xinge Zhu,et al.  Cylinder3D: An Effective 3D Framework for Driving-scene LiDAR Semantic Segmentation , 2020, ArXiv.

[45]  Jiyu Cheng,et al.  Robust Semantic Mapping in Challenging Environments , 2019, Robotica.

[46]  Jörg Stückler,et al.  Scene flow propagation for semantic mapping and object discovery in dynamic street scenes , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[47]  Hannes Sommer,et al.  SegMap: Segment-based mapping and localization using data-driven descriptors , 2019, Int. J. Robotics Res..

[48]  Jonathan P. How,et al.  Multi-Robot Distributed Semantic Mapping in Unfamiliar Environments through Online Matching of Learned Representations , 2021, 2021 IEEE International Conference on Robotics and Automation (ICRA).

[49]  Roland Siegwart,et al.  Volumetric Instance-Aware Semantic Mapping and 3D Object Discovery , 2019, IEEE Robotics and Automation Letters.

[50]  Stefan Leutenegger,et al.  Efficient Octree-Based Volumetric SLAM Supporting Signed-Distance and Occupancy Mapping , 2018, IEEE Robotics and Automation Letters.

[51]  Ming Liu,et al.  Virtual-to-real deep reinforcement learning: Continuous control of mobile robots for mapless navigation , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[52]  Marc Pollefeys,et al.  Robust Dense Mapping for Large-Scale Dynamic Environments , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[53]  Cyrill Stachniss,et al.  SuMa++: Efficient LiDAR-based Semantic SLAM , 2019, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[54]  Fabio Tozeto Ramos,et al.  A Sparse Covariance Function for Exact Gaussian Process Inference in Large Datasets , 2009, IJCAI.

[55]  Alexey Dosovitskiy,et al.  End-to-End Driving Via Conditional Imitation Learning , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[56]  Ziqi Zhang,et al.  Detect-SLAM: Making Object Detection and SLAM Mutually Beneficial , 2018, 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).

[57]  Yong Jae Lee,et al.  HPLFlowNet: Hierarchical Permutohedral Lattice FlowNet for Scene Flow Estimation on Large-Scale Point Clouds , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[58]  Linyan Cui,et al.  SOF-SLAM: A Semantic Visual SLAM for Dynamic Environments , 2019, IEEE Access.

[59]  Cyrill Stachniss,et al.  Efficient Surfel-Based SLAM using 3D Laser Range Data in Urban Environments , 2018, Robotics: Science and Systems.

[60]  Qi Wei,et al.  DS-SLAM: A Semantic Visual SLAM towards Dynamic Environments , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[61]  Dimitris N. Metaxas,et al.  MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird’s Eye View Maps , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[62]  Cyrill Stachniss,et al.  RangeNet ++: Fast and Accurate LiDAR Semantic Segmentation , 2019, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[63]  He Xu,et al.  DOS-SLAM: A Real-time Dynamic Object Segmentation Visual SLAM System , 2019, ACAI 2019.

[64]  Binbin Xu,et al.  MID-Fusion: Octree-based Object-Level Multi-Instance Dynamic SLAM , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[65]  Roger Woodard,et al.  Interpolation of Spatial Data: Some Theory for Kriging , 1999, Technometrics.

[66]  Brian Okorn,et al.  Just Go With the Flow: Self-Supervised Scene Flow Estimation , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).