Bidirectional Long Short-Term Memory Network for Vehicle Behavior Recognition

Vehicle behavior recognition is an attractive research field which is useful for many computer vision and intelligent traffic analysis tasks. This paper presents an all-in-one behavior recognition framework for moving vehicles based on the latest deep learning techniques. Unlike traditional traffic analysis methods which rely on low-resolution videos captured by road cameras, we capture 4K ( 3840 × 2178 ) traffic videos at a busy road intersection of a modern megacity by flying a unmanned aerial vehicle (UAV) during the rush hours. We then manually annotate locations and types of road vehicles. The proposed method consists of the following three steps: (1) vehicle detection and type recognition based on deep neural networks; (2) vehicle tracking by data association and vehicle trajectory modeling; (3) vehicle behavior recognition by nearest neighbor search and by bidirectional long short-term memory network, respectively. This paper also presents experimental results of the proposed framework in comparison with state-of-the-art approaches on the 4K testing traffic video, which demonstrated the effectiveness and superiority of the proposed method.

[1]  Patrick L. Lawrence,et al.  Unmanned Systems and Managing from Above: The Practical Implications of UAVs for Research Applications Addressing Urban Sustainability , 2016 .

[2]  Wesam A. Sakla,et al.  A Large Contextual Dataset for Classification, Detection and Counting of Cars with Deep Learning , 2016, ECCV.

[3]  Farid Melgani,et al.  Automatic Car Counting Method for Unmanned Aerial Vehicle Images , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[4]  Kaiming He,et al.  Focal Loss for Dense Object Detection , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[5]  A. Puri.  A Survey of Unmanned Aerial Vehicles ( UAV ) for Traffic Surveillance , 2005 .

[6]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[7]  Venkatesh Saligrama,et al.  Video Anomaly Identification , 2010, IEEE Signal Processing Magazine.

[8]  Paola Mello,et al.  Image analysis and rule-based reasoning for a traffic monitoring system , 1999, Proceedings 199 IEEE/IEEJ/JSAI International Conference on Intelligent Transportation Systems (Cat. No.99TH8383).

[9]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[10]  Kuk-Jin Yoon,et al.  Robust Online Multi-object Tracking Based on Tracklet Confidence and Online Discriminative Appearance Learning , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Christopher D. Manning,et al.  Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks , 2015, ACL.

[12]  Wolfgang Rosenstiel,et al.  Object-Oriented Bayesian Networks for Detection of Lane Change Maneuvers , 2012, IEEE Intelligent Transportation Systems Magazine.

[13]  Silvio Savarese,et al.  Learning to Track: Online Multi-object Tracking by Decision Making , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[14]  Giuseppe Salvo,et al.  Urban traffic analysis through an UAV , 2014 .

[15]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[17]  Zhaoquan Cai,et al.  Automatic Detection of Vehicle Activities Based on Particle Filter Tracking , 2009 .

[18]  Peng Zhang,et al.  Vehicle Behavior Analysis Using Target Motion Trajectories , 2014, IEEE Transactions on Vehicular Technology.

[19]  Thomas Oommen,et al.  Evaluating the Use of Unmanned Aerial Vehicles for Transportation Purposes , 2015 .

[20]  Kaiming He,et al.  Feature Pyramid Networks for Object Detection , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Jitendra Malik,et al.  Fast vehicle detection with probabilistic feature grouping and its application to vehicle tracking , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[22]  Bryan E. Porter,et al.  Predicting Red-Light Running Behavior: A Traffic Safety Study in Three Urban Settings , 2000 .

[23]  Dot Hs,et al.  Crash Factors in Intersection-Related Crashes: An On-Scene Perspective , 2010 .

[24]  Jiamin Wu,et al.  基于二次谱聚类和HMM-RF混合模型的车辆行为识别方法研究 (Vehicle Behavior Recognition Method Based on Quadratic Spectral Clustering and HMM-RF Hybrid Model) , 2016, 计算机科学.

[25]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[26]  Mark R. McCord,et al.  Roadway traffic monitoring from an unmanned aerial vehicle , 2006 .

[27]  Uwe Stilla,et al.  Comparison of Two Methods for Vehicle Extraction From Airborne LiDAR Data Toward Motion Analysis , 2011, IEEE Geoscience and Remote Sensing Letters.

[28]  Ken'ichi Kamijo,et al.  Stock price pattern recognition-a recurrent neural network approach , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[29]  Mohan M. Trivedi,et al.  Trajectory Learning for Activity Understanding: Unsupervised, Multilevel, and Long-Term Adaptive Approach , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Mohan M. Trivedi,et al.  A Survey of Vision-Based Trajectory Learning and Analysis for Surveillance , 2008, IEEE Transactions on Circuits and Systems for Video Technology.

[31]  Wei Shi,et al.  Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification , 2016, ACL.

[32]  Nikolaos Papanikolopoulos,et al.  Learning to Recognize Video-Based Spatiotemporal Events , 2009, IEEE Transactions on Intelligent Transportation Systems.

[33]  Tieniu Tan,et al.  Traffic accident prediction using vehicle tracking and trajectory analysis , 2003, Proceedings of the 2003 IEEE International Conference on Intelligent Transportation Systems.

[34]  周鑫,et al.  Tracking-learning-detection (TLD)-based video object tracking method , 2012 .

[35]  M. Bernardine Dias,et al.  The Dynamic Hungarian Algorithm for the Assignment Problem with Changing Costs , 2007 .

[36]  Fangliang Chen,et al.  Detecting and tracking vehicles in traffic by unmanned aerial vehicles , 2016 .

[37]  Florent Altché,et al.  An LSTM network for highway trajectory prediction , 2017, 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC).

[38]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[39]  Peter Reinartz,et al.  An Operational System for Estimating Road Traffic Information from Aerial Images , 2014, Remote. Sens..

[40]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[41]  James W. Davis,et al.  The Recognition of Human Movement Using Temporal Templates , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[42]  S. Hinz,et al.  Detection and counting of cars in aerial images , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[43]  Saeid Nahavandi,et al.  Efficient Road Detection and Tracking for Unmanned Aerial Vehicle , 2015, IEEE Transactions on Intelligent Transportation Systems.

[44]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[45]  Konstantinos Kanistras,et al.  A survey of unmanned aerial vehicles (UAVs) for traffic monitoring , 2013, 2013 International Conference on Unmanned Aircraft Systems (ICUAS).

[46]  Nikolaos Papanikolopoulos,et al.  Clustering of Vehicle Trajectories , 2010, IEEE Transactions on Intelligent Transportation Systems.

[47]  Yaoqi Zhou,et al.  Improving protein disorder prediction by deep bidirectional long short‐term memory recurrent neural networks , 2016, Bioinform..

[48]  Peter Schallauer,et al.  Multimodal highway monitoring for robust incident detection , 2010, 13th International IEEE Conference on Intelligent Transportation Systems.

[49]  Dumitru Erhan,et al.  Scalable Object Detection Using Deep Neural Networks , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[50]  Farid Melgani,et al.  Detecting Cars in UAV Images With a Catalog-Based Approach , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[51]  Alexander Barth,et al.  Estimating the Driving State of Oncoming Vehicles From a Moving Platform Using Stereo Vision , 2009, IEEE Transactions on Intelligent Transportation Systems.

[52]  Fredrik Gustafsson,et al.  Road Target Search and Tracking with Gimballed Vision Sensor on an Unmanned Aerial Vehicle , 2012, Remote. Sens..

[53]  Luc Van Gool,et al.  Efficient Non-Maximum Suppression , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[54]  Davy Janssens,et al.  th EURO Working Group on Transportation Meeting , EWGT 2016 , 5-7 September 2016 , Istanbul , Turkey UAV-Based Traffic Analysis : A Universal Guiding Framework Based on Literature Survey , 2017 .

[55]  B. Coifman,et al.  Surface Transportation Surveillance from Unmanned Aerial Vehicles , 2003 .

[56]  Ali Farhadi,et al.  YOLOv3: An Incremental Improvement , 2018, ArXiv.

[57]  Patrick Bouthemy,et al.  A HMM-Based Method for Recognizing Dynamic Video Contents from Trajectories , 2007, 2007 IEEE International Conference on Image Processing.