Sequential Point Cloud Prediction in Interactive Scenarios: A Survey

Point cloud has been widely used in the field of autonomous driving since it can provide a more comprehensive three-dimensional representation of the environment than 2D images. Point-wise prediction based on point cloud sequence (PCS) is an essential part of environment understanding, which can assist in the decision-making and motion-planning of autonomous vehicles. However, PCS prediction has not been deeply researched in the literature. This paper proposes a brief review of the sequential point cloud prediction methods, focusing on interactive scenarios. Firstly, we define the PCS prediction problem and introduce commonly-used frameworks. Secondly, by reviewing non-predictive problems, we analyze and summarize the spatio-temporal feature extraction methods based on PCS. On this basis, we review two types of PCS prediction tasks, scene flow estimation (SFE) and point cloud location prediction (PCLP), highlighting their connections and differences. Finally, we discuss some opening issues and point out some potential research directions. Keywords—point cloud, autonomous driving, environmental understanding, scene flow

[1]  Edwin Olson,et al.  Multipolicy decision-making for autonomous driving via changepoint-based behavior prediction: Theory and experiment , 2015, Autonomous Robots.

[2]  Koichi Hashimoto,et al.  Spatiotemporal Learning of Dynamic Gestures from 3D Point Cloud Data , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[3]  Hesheng Wang,et al.  Anchor-Based Spatio-Temporal Attention 3-D Convolutional Networks for Dynamic 3-D Point Cloud Sequences , 2020, IEEE Transactions on Instrumentation and Measurement.

[4]  Marco Fiore,et al.  CloudLSTM: A Recurrent Neural Model for Spatiotemporal Point-cloud Stream Forecasting , 2019, AAAI.

[5]  Zhuwen Li,et al.  PointPWC-Net: A Coarse-to-Fine Network for Supervised and Self-Supervised Scene Flow Estimation on 3D Point Clouds , 2019, ArXiv.

[6]  Qifeng Chen,et al.  TPCN: Temporal Point Cloud Networks for Motion Forecasting , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Yong Jae Lee,et al.  HPLFlowNet: Hierarchical Permutohedral Lattice FlowNet for Scene Flow Estimation on Large-Scale Point Clouds , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Hao Wang,et al.  SpSequenceNet: Semantic Segmentation Network on 4D Point Clouds , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Ting Hu,et al.  Deep Learning on Point Clouds and Its Application: A Survey , 2019, Sensors.

[10]  Bin Yang,et al.  Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[11]  M. Tomizuka,et al.  EvolveGraph: Multi-Agent Trajectory Prediction with Dynamic Relational Reasoning , 2020, NeurIPS.

[12]  Yingli Tian,et al.  Self-supervised 4D Spatio-temporal Feature Learning via Order Prediction of Sequential Point Cloud Clips , 2021, 2021 IEEE Winter Conference on Applications of Computer Vision (WACV).

[13]  Avideh Zakhor,et al.  Temporal LiDAR Frame Prediction for Autonomous Driving , 2020, 2020 International Conference on 3D Vision (3DV).

[14]  Yinlong Liu,et al.  MoNet: Motion-Based Point Cloud Prediction Network , 2020, IEEE Transactions on Intelligent Transportation Systems.

[15]  Yi Yang,et al.  PointRNN: Point Recurrent Neural Network for Moving Point Cloud Processing , 2019, ArXiv.

[16]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Stefan Jeschke,et al.  Tranquil Clouds: Neural Networks for Learning Temporally Coherent Features in Point Clouds , 2019, ICLR.

[18]  Avideh Zakhor,et al.  3d Object Detection For Autonomous Driving Using Temporal Lidar Data , 2020, 2020 IEEE International Conference on Image Processing (ICIP).

[19]  Mohammed Bennamoun,et al.  Deep Learning for 3D Point Clouds: A Survey , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Myoungho Sunwoo,et al.  Hybrid Trajectory Planning for Autonomous Driving in On-Road Dynamic Scenarios , 2019, IEEE Transactions on Intelligent Transportation Systems.

[21]  Dimitris N. Metaxas,et al.  MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird’s Eye View Maps , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Leonidas J. Guibas,et al.  CaSPR: Learning Canonical Spatiotemporal Point Cloud Representations , 2020, NeurIPS.

[23]  You Li,et al.  Lidar for Autonomous Driving: The Principles, Challenges, and Trends for Automotive Lidar and Perception Systems , 2020, IEEE Signal Processing Magazine.

[24]  Mohamed Zahran,et al.  YOLO4D: A Spatio-temporal Approach for Real-time Multi-object Detection and Classification from LiDAR Point Clouds , 2018 .

[25]  Ruigang Yang,et al.  LiDAR-Based Online 3D Video Object Detection With Graph-Based Message Passing and Spatiotemporal Transformer Attention , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Xiaogang Wang,et al.  Shape2Motion: Joint Analysis of Motion Parts and Attributes From 3D Shapes , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Alexandre Boulch,et al.  FLOT: Scene Flow on Point Clouds Guided by Optimal Transport , 2020, ECCV.

[28]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[29]  Silvia Rossi,et al.  Spatio-Temporal Graph-RNN for Point Cloud Prediction , 2021, 2021 IEEE International Conference on Image Processing (ICIP).

[30]  Jeannette Bohg,et al.  MeteorNet: Deep Learning on Dynamic 3D Point Cloud Sequences , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[31]  Yu Zhang,et al.  Hybrid Trajectory Planning for Autonomous Driving in Highly Constrained Environments , 2018, IEEE Access.

[32]  Masayoshi Tomizuka,et al.  Conditional Generative Neural System for Probabilistic Trajectory Prediction , 2019, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[33]  J. Beyerer,et al.  LiDAR-based Recurrent 3D Semantic Segmentation with Temporal Memory Alignment , 2020, 2020 International Conference on 3D Vision (3DV).

[34]  Dong Tian,et al.  FESTA: Flow Estimation via Spatial-Temporal Attention for Scene Point Clouds , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Leonidas J. Guibas,et al.  FlowNet3D: Learning Scene Flow in 3D Point Clouds , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Andry Rakotonirainy,et al.  Perception, information processing and modeling: Critical stages for autonomous driving applications , 2017, Annu. Rev. Control..

[37]  Bin Zhou,et al.  Self‐Supervised Learning of Part Mobility from Point Cloud Sequence , 2020, Comput. Graph. Forum.

[38]  Leonidas J. Guibas,et al.  PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space , 2017, NIPS.

[39]  Marco Pavone,et al.  The Trajectron: Probabilistic Multi-Agent Trajectory Modeling With Dynamic Spatiotemporal Graphs , 2018, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[40]  Hui Huang,et al.  RPM-Net , 2019, ACM Trans. Graph..

[41]  David Isele,et al.  Reinforcement Learning for Autonomous Driving with Latent State Inference and Spatial-Temporal Relationships , 2020, 2021 IEEE International Conference on Robotics and Automation (ICRA).

[42]  Andrei Furda,et al.  Enabling Safe Autonomous Driving in Real-World City Traffic Using Multiple Criteria Decision Making , 2011, IEEE Intelligent Transportation Systems Magazine.

[43]  Mario Zanon,et al.  Real-Time Constrained Trajectory Planning and Vehicle Control for Proactive Autonomous Driving With Road Users , 2019, 2019 18th European Control Conference (ECC).

[44]  Ryan M. Eustice,et al.  A learning approach for real-time temporal scene flow estimation from LIDAR data , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[45]  Xilin Chen,et al.  An Efficient PointLSTM for Point Clouds Based Gesture Recognition , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  Christoph Stiller,et al.  Decision making for autonomous driving considering interaction and uncertain prediction of surrounding vehicles , 2017, 2017 IEEE Intelligent Vehicles Symposium (IV).

[47]  Jianren Wang,et al.  Inverting the Pose Forecasting Pipeline with SPF2: Sequential Pointcloud Forecasting for Sequential Pose Forecasting , 2020, CoRL.

[48]  Brian Okorn,et al.  Just Go With the Flow: Self-Supervised Scene Flow Estimation , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[49]  Wolfram Burgard,et al.  Rigid scene flow for 3D LiDAR scans , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[50]  Silvio Savarese,et al.  4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[51]  Amir Rasouli,et al.  Deep Learning for Vision-based Prediction: A Survey , 2020, ArXiv.

[52]  Weiwen Deng,et al.  Dynamic Trajectory Planning for Vehicle Autonomous Driving , 2013, 2013 IEEE International Conference on Systems, Man, and Cybernetics.

[53]  Sergio Orts-Escolano,et al.  A Review on Deep Learning Techniques for Video Prediction , 2020, IEEE transactions on pattern analysis and machine intelligence.

[54]  Aseem Behl,et al.  PointFlowNet: Learning Representations for Rigid Motion Estimation From Point Clouds , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[55]  Stephan Sigg,et al.  Motion Pattern Recognition in 4D Point Clouds , 2020, 2020 IEEE 30th International Workshop on Machine Learning for Signal Processing (MLSP).

[56]  Silvio Savarese,et al.  Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks , 2019, NeurIPS.

[57]  Silvio Savarese,et al.  Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[58]  Peter V. Gehler,et al.  Learning Sparse High Dimensional Filters: Image Filtering, Dense CRFs and Bilateral Neural Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[59]  Leonidas J. Guibas,et al.  Weakly Supervised Learning of Rigid 3D Scene Flow , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[60]  Yin Zhou,et al.  VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[61]  Thomas Funkhouser,et al.  An LSTM Approach to Temporal 3D Object Detection in LiDAR Point Clouds , 2020, ECCV.

[62]  V. Prisacariu,et al.  FlowNet3D++: Geometric Losses For Deep Scene Flow Estimation , 2019, 2020 IEEE Winter Conference on Applications of Computer Vision (WACV).