Handling pedestrians in self-driving cars using image tracking and alternative path generation with Frenét frames

Abstract The development of intelligent autonomous cars is of great interest. A particular and challenging problem is to handle pedestrians, for example, crossing or walking along the road. Since pedestrians are one of the most fragile elements in traffic, a reliable pedestrian detection and handling system is mandatory. The current pedestrian handling system of our autonomous cars suffers from the limitation of the pure detection-based systems, i.e., it limits the autonomous car system to make decisions based only on the very present moment. This work improves the pedestrian handling systems by incorporating an object tracker with the aim of predicting the pedestrian’s behavior. With this knowledge, the autonomous car can better decide the time to stop and to start moving, providing a more comfortable, efficient, and safer driving experience. The proposed method was augmented with a path generator, based on Frenet Frames, and incorporated to our self-driving car in order to enable a better decision making and to enable overtaking pedestrians. The behaviour of our self-driving car was evaluated in both simulated and real-world scenarios. Results showed the proposed system is safer and more efficient than the system without tracking functionality due to the early decision capability.

[1]  Yaser Sheikh,et al.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Konrad Schindler,et al.  Learning by Tracking: Siamese CNN for Robust Target Association , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[3]  Alberto Ferreira de Souza,et al.  Automatic large-scale data acquisition via crowdsourcing for crosswalk classification: A deep learning approach , 2017, Comput. Graph..

[4]  Cristiano Premebida,et al.  Exploiting LIDAR-based features on pedestrian detection in urban scenarios , 2009, 2009 12th International IEEE Conference on Intelligent Transportation Systems.

[5]  Martial Hebert,et al.  The Pose Knows: Video Forecasting by Generating Pose Futures , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[6]  Jonathan Tompson,et al.  Towards Accurate Multi-person Pose Estimation in the Wild , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  George Papandreou,et al.  Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation , 2018, ECCV.

[8]  Edilson de Aguiar,et al.  Large-scale mapping in complex field scenarios using an autonomous car , 2016, Expert Syst. Appl..

[9]  Jitendra Malik,et al.  Recurrent Network Models for Human Dynamics , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[10]  Xiaohui Li,et al.  Real-Time Trajectory Planning for Autonomous Urban Driving: Framework, Algorithms, and Verifications , 2016, IEEE/ASME Transactions on Mechatronics.

[11]  Bodo Rosenhahn,et al.  Fusion of Head and Full-Body Detectors for Multi-object Tracking , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[12]  Long Chen,et al.  Real-Time Multiple People Tracking with Deeply Learned Candidate Selection and Person Re-Identification , 2018, 2018 IEEE International Conference on Multimedia and Expo (ICME).

[13]  Alberto Ferreira de Souza,et al.  Neural-based model predictive control for tackling steering delays of autonomous cars , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[14]  Sebastian Thrun,et al.  Probabilistic robotics , 2002, CACM.

[15]  Julius Ziegler,et al.  Optimal trajectory generation for dynamic street scenarios in a Frenét Frame , 2010, 2010 IEEE International Conference on Robotics and Automation.

[16]  Yi Li,et al.  R-FCN: Object Detection via Region-based Fully Convolutional Networks , 2016, NIPS.

[17]  Heng Wang,et al.  Robotics and Autonomous Systems , 2022 .

[18]  Rodrigo F. Berriel,et al.  Deep Learning-Based Large-Scale Automatic Satellite Crosswalk Classification , 2017, IEEE Geoscience and Remote Sensing Letters.

[19]  Geng Yan,et al.  Real-time Detection, Tracking, and Classification of Moving and Stationary Objects using Multiple Fisheye Images , 2018, 2018 IEEE Intelligent Vehicles Symposium (IV).

[20]  Edilson de Aguiar,et al.  Ego-Lane Analysis System (ELAS): Dataset and algorithms , 2017, Image Vis. Comput..

[21]  Edilson de Aguiar,et al.  Traffic sign detection with VG-RAM weightless neural networks , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[22]  Michael J. Black,et al.  On Human Motion Prediction Using Recurrent Neural Networks , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Yang Zhang,et al.  Heterogeneous Association Graph Fusion for Target Association in Multiple Object Tracking , 2019, IEEE Transactions on Circuits and Systems for Video Technology.

[24]  Thiago Oliveira-Santos,et al.  Handling Pedestrians in Crosswalks Using Deep Neural Networks in the IARA Autonomous Car , 2018, 2018 International Joint Conference on Neural Networks (IJCNN).

[25]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[26]  Ross B. Girshick,et al.  Mask R-CNN , 2017, 1703.06870.

[27]  Miguel Angel Sotelo,et al.  Autonomous Navigation and Obstacle Avoidance of a Micro-Bus , 2013 .

[28]  Danica Kragic,et al.  Deep Representation Learning for Human Motion Prediction and Classification , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Alberto Ferreira de Souza,et al.  A simple yet effective obstacle avoider for the IARA autonomous car , 2016, 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC).

[30]  Edilson de Aguiar,et al.  A light-weight yet accurate localization system for autonomous cars in large-scale and complex environments , 2016, 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC).

[31]  Ali Farhadi,et al.  YOLO9000: Better, Faster, Stronger , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Victor M. Becerra,et al.  Modeling, Simulation, and Control of Pedestrian Avoidance Maneuver for an Urban Electric Vehicle , 2015, 2015 IEEE European Modelling Symposium (EMS).

[33]  Luc Van Gool,et al.  Pedestrian detection at 100 frames per second , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[34]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[35]  Shuicheng Yan,et al.  Scale-Aware Fast R-CNN for Pedestrian Detection , 2015, IEEE Transactions on Multimedia.

[36]  Alberto Ferreira de Souza,et al.  A Model-Predictive Motion Planner for the IARA autonomous car , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[37]  Jingdong Wang,et al.  Deeply-Learned Part-Aligned Representations for Person Re-identification , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).