Video action recognition for lane-change classification and prediction of surrounding vehicles

In highway scenarios, an alert human driver will typically anticipate early cut-in/cut-out maneuvers of surrounding vehicles using visual cues mainly. Autonomous vehicles must anticipate these situations at an early stage too, to increase their safety and efficiency. In this work, lane-change recognition and prediction tasks are posed as video action recognition problems. Up to four different two-stream-based approaches, that have been successfully applied to address human action recognition, are adapted here by stacking visual cues from forward-looking video cameras to recognize and anticipate lane-changes of target vehicles. We study the influence of context and observation horizons on performance, and different prediction horizons are analyzed. The different models are trained and evaluated using the PREVENTION dataset. The obtained results clearly demonstrate the potential of these methodologies to serve as robust predictors of future lane-changes of surrounding vehicles proving an accuracy higher than 90% in time horizons of between 1-2 seconds.

[1]  Dongsuk Kum,et al.  The multilayer perceptron approach to lateral motion prediction of surrounding vehicles for autonomous vehicles , 2016, 2016 IEEE Intelligent Vehicles Symposium (IV).

[2]  Klaus C. J. Dietmayer,et al.  A learning concept for behavior prediction in traffic situations , 2013, 2013 IEEE Intelligent Vehicles Symposium (IV).

[3]  Ruigang Yang,et al.  The ApolloScape Open Dataset for Autonomous Driving and Its Application , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Wolfgang Rosenstiel,et al.  Object-oriented Bayesian networks for detection of lane change maneuvers , 2011, 2011 IEEE Intelligent Vehicles Symposium (IV).

[5]  Andreas Lawitzky,et al.  A Combined Model- and Learning-Based Framework for Interaction-Aware Maneuver Prediction , 2016, IEEE Transactions on Intelligent Transportation Systems.

[6]  Jitendra Malik,et al.  SlowFast Networks for Video Recognition , 2018, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[7]  Mohan M. Trivedi,et al.  Looking at Vehicles on the Road: A Survey of Vision-Based Vehicle Detection, Tracking, and Behavior Analysis , 2013, IEEE Transactions on Intelligent Transportation Systems.

[8]  Ignacio Parra,et al.  Experimental validation of lane-change intention prediction methodologies based on CNN and LSTM , 2019, 2019 IEEE Intelligent Transportation Systems Conference (ITSC).

[9]  Jason J. Corso,et al.  Predicting Future Lane Changes of Other Highway Vehicles using RNN-based Deep Models , 2018, ArXiv.

[10]  Ignacio Parra,et al.  Two-camera based accurate vehicle speed measurement using average speed at a fixed point , 2016, 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC).

[11]  Andrew Zisserman,et al.  Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Ümit Özgüner,et al.  Trajectory prediction of a lane changing vehicle based on driver behavior estimation and classification , 2014, 17th International IEEE Conference on Intelligent Transportation Systems (ITSC).

[13]  Gunnar Farnebäck,et al.  Two-Frame Motion Estimation Based on Polynomial Expansion , 2003, SCIA.

[14]  Richard P. Wildes,et al.  Spatiotemporal Multiplier Networks for Video Action Recognition , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Huijun Di,et al.  Manifold Learning for Lane-changing Behavior Recognition in Urban Traffic , 2019, 2019 IEEE Intelligent Transportation Systems Conference (ITSC).

[16]  Mohan M. Trivedi,et al.  Convolutional Social Pooling for Vehicle Trajectory Prediction , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[17]  Mohan M. Trivedi,et al.  Video-based lane estimation and tracking for driver assistance: survey, system, and evaluation , 2006, IEEE Transactions on Intelligent Transportation Systems.

[18]  Andrew Zisserman,et al.  Two-Stream Convolutional Networks for Action Recognition in Videos , 2014, NIPS.

[19]  Dizan Vasquez,et al.  A survey on motion prediction and risk assessment for intelligent vehicles , 2014, ROBOMECH Journal.

[20]  Cordelia Schmid,et al.  AVA: A Video Dataset of Spatio-Temporally Localized Atomic Visual Actions , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[21]  Sara McMains,et al.  Convolution neural network-based lane change intention prediction of surrounding vehicles for ACC , 2017, 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC).

[22]  Ignacio Parra,et al.  The PREVENTION dataset: a novel benchmark for PREdiction of VEhicles iNTentIONs , 2019, 2019 IEEE Intelligent Transportation Systems Conference (ITSC).

[23]  Junxiang Li,et al.  A Dynamic Bayesian Network for Vehicle Maneuver Prediction in Highway Driving Scenarios: Framework and Verification , 2019, Electronics.

[24]  Torsten Bertram,et al.  Probabilistic Lane Change Prediction using Gaussian Process Neural Networks , 2019, 2019 IEEE Intelligent Transportation Systems Conference (ITSC).

[25]  Klaus-Dieter Kuhnert,et al.  When will it change the lane? A probabilistic regression approach for rarely occurring events , 2015, 2015 IEEE Intelligent Vehicles Symposium (IV).

[26]  Klaus-Dieter Kuhnert,et al.  A lane change detection approach using feature ranking with maximized predictive power , 2014, 2014 IEEE Intelligent Vehicles Symposium Proceedings.

[27]  The PREVENTION Challenge: How Good Are Humans Predicting Lane Changes? , 2020, 2020 IEEE Intelligent Vehicles Symposium (IV).

[28]  John K. Tsotsos,et al.  Two-Stream Networks for Lane-Change Prediction of Surrounding Vehicles , 2020, 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC).

[29]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Miguel Ángel Sotelo,et al.  The Experience of DRIVERTIVE-DRIVERless cooperaTIve VEhicle-Team in the 2016 GCDC , 2018, IEEE Transactions on Intelligent Transportation Systems.

[31]  Yun Fu,et al.  Human Action Recognition and Prediction: A Survey , 2018, International Journal of Computer Vision.

[32]  Feng Gao,et al.  A comprehensive review of the development of adaptive cruise control systems , 2010 .

[33]  Ignacio Parra,et al.  Performance analysis of Vehicle-to-Vehicle communications for critical tasks in autonomous driving , 2019, 2019 IEEE Intelligent Transportation Systems Conference (ITSC).

[34]  Mohan M. Trivedi,et al.  Multi-Modal Trajectory Prediction of Surrounding Vehicles with Maneuver based LSTMs , 2018, 2018 IEEE Intelligent Vehicles Symposium (IV).

[35]  Huijing Zhao,et al.  On-Road Vehicle Trajectory Collection and Scene-Based Lane Change Analysis: Part I , 2017, IEEE Transactions on Intelligent Transportation Systems.

[36]  Lutz Eckstein,et al.  The highD Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systems , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[37]  Ignacio Parra,et al.  Vehicle trajectory and lane change prediction using ANN and SVM classifiers , 2017, 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC).

[38]  Huijing Zhao,et al.  On-Road Vehicle Trajectory Collection and Scene-Based Lane Change Analysis: Part II , 2017, IEEE Trans. Intell. Transp. Syst..

[39]  Masayoshi Tomizuka,et al.  INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps , 2019, ArXiv.