Incorporating Orientations into End-to-end Driving Model for Steering Control

In this paper, we present a novel end-to-end deep neural network model for autonomous driving that takes monocular image sequence as input, and directly generates the steering control angle. Firstly, we model the end-to-end driving problem as a local path planning process. Inspired by the environmental representation in the classical planning algorithms(i.e. the beam curvature method), pixel-wise orientations are fed into the network to learn direction-aware features. Next, to handle the imbalanced distribution of steering values in training datasets, we propose an improvement on a cost-sensitive loss function named SteeringLoss2. Besides, we also present a new end-to-end driving dataset, which provides corresponding LiDAR and image sequences, as well as standard driving behaviors. Our dataset includes multiple driving scenarios, such as urban, country, and off-road. Numerous experiments are conducted on both public available LiVi-Set and our own dataset, and the results show that the model using our proposed methods can predict steering angle accurately.

[1]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[2]  Dongpu Cao,et al.  End-to-End Autonomous Driving: An Angle Branched Network Approach , 2019, IEEE Transactions on Vehicular Technology.

[3]  Yadong Mu,et al.  Learning End-to-End Autonomous Steering Model from Spatial and Temporal Visual Cues , 2017, VSCC '17.

[4]  Yann LeCun,et al.  Off-Road Obstacle Avoidance through End-to-End Learning , 2005, NIPS.

[5]  Trevor Darrell,et al.  BDD100K: A Diverse Driving Video Database with Scalable Annotation Tooling , 2018, ArXiv.

[6]  Jianxiong Xiao,et al.  DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[7]  Vidya N. Murali,et al.  DeepLanes: End-To-End Lane Position Estimation Using Deep Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[8]  Sridha Sridharan,et al.  Going Deeper: Autonomous Steering with Neural Memory Networks , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[9]  Mariusz Bojarski,et al.  VisualBackProp: Efficient Visualization of CNNs for Autonomous Driving , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[10]  Dan Wang,et al.  End-to-End Self-Driving Using Deep Neural Networks with Multi-auxiliary Tasks , 2019, Automotive Innovation.

[11]  Jiebo Luo,et al.  End-to-end Multi-Modal Multi-Task Vehicle Control for Self-Driving Cars with Visual Perceptions , 2018, 2018 24th International Conference on Pattern Recognition (ICPR).

[12]  Ming Yang,et al.  SteeringLoss: A Cost-Sensitive Loss Function for the End-to-End Steering Estimation , 2021, IEEE Transactions on Intelligent Transportation Systems.

[13]  Kay Chen Tan,et al.  Evolutionary artificial potential fields and their application in real time robot path planning , 2000, Proceedings of the 2000 Congress on Evolutionary Computation. CEC00 (Cat. No.00TH8512).

[14]  Xin Zhang,et al.  End to End Learning for Self-Driving Cars , 2016, ArXiv.

[15]  Cewu Lu,et al.  LiDAR-Video Driving Dataset: Learning Driving Policies Effectively , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[16]  Kaiming He,et al.  Focal Loss for Dense Object Detection , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[17]  John F. Canny,et al.  Interpretable Learning for Self-Driving Cars by Visualizing Causal Attention , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[18]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[19]  Yang Gao,et al.  End-to-End Learning of Driving Models from Large-Scale Video Datasets , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Ross B. Girshick,et al.  Focal Loss for Dense Object Detection , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  J. López,et al.  A new approach to local navigation for autonomous driving vehicles based on the curvature velocity method , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[22]  Wenshuo Wang,et al.  Feature analysis and selection for training an end-to-end autonomous vehicle controller using deep learning approach , 2017, 2017 IEEE Intelligent Vehicles Symposium (IV).

[23]  Joaquín Lopez Fernández,et al.  Improving collision avoidance for mobile robots in partially known environments: the beam curvature method , 2004, Robotics Auton. Syst..

[24]  Jiman Kim,et al.  End-To-End Ego Lane Estimation Based on Sequential Transfer Learning for Self-Driving Cars , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[25]  Lawrence D. Jackel,et al.  Explaining How a Deep Neural Network Trained with End-to-End Learning Steers a Car , 2017, ArXiv.

[26]  Dean Pomerleau,et al.  ALVINN, an autonomous land vehicle in a neural network , 2015 .