Learning to tell brake and turn signals in videos using CNN-LSTM structure

We present a method that learns to tell rear signals from a number of frames using a deep learning framework. The proposed framework extracts spatial features with a convolution neural network (CNN), and then applies a long short term memory (LSTM) network to learn the long-term dependencies. The brake signal classifier is trained using RGB frames, while the turn signal is recognized via a two-step localization approach. The two separate classifiers are learned to recognize the static brake signals and the dynamic turn signals. As a result, our recognition system can recognize 8 different rear signals via the combined two classifiers in real-world traffic scenes. Experimental results show that our method is able to obtain more accurate predictions than using only the CNN to classify rear signals with time sequence inputs.

[1]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[2]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[3]  Poj Tangamchit,et al.  Predictive brake warning at night using taillight characteristic , 2009, 2009 IEEE International Symposium on Industrial Electronics.

[4]  Antonio Torralba,et al.  SIFT Flow: Dense Correspondence across Scenes and Its Applications , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Senem Velipasalar,et al.  A Robust Algorithm for the Detection of Vehicle Turn Signals and Brake Lights , 2012, 2012 IEEE Ninth International Conference on Advanced Video and Signal-Based Surveillance.

[6]  Yi Li,et al.  Recognition algorithm for turn light of front vehicle , 2012 .

[7]  Mubarak Shah,et al.  UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild , 2012, ArXiv.

[8]  Senem Velipasalar,et al.  Autonomous tracking of vehicle rear lights and detection of brakes and turn signals , 2012, 2012 IEEE Symposium on Computational Intelligence for Security and Defence Applications.

[9]  Wojciech Zaremba,et al.  Learning to Execute , 2014, ArXiv.

[10]  Jun-Wei Hsieh,et al.  Nighttime Turn Signal Detection by Scatter Modeling and Reflectance-Based Direction Recognition , 2014, IEEE Sensors Journal.

[11]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[12]  Markus Enzweiler,et al.  Will this car change the lane? - Turn signal recognition in the frequency domain , 2014, 2014 IEEE Intelligent Vehicles Symposium Proceedings.

[13]  Hsin-Mu Tsai,et al.  A Vision-Based Hierarchical Framework for Autonomous Front-Vehicle Taillights Detection and Signal Recognition , 2015, 2015 IEEE 18th International Conference on Intelligent Transportation Systems.

[14]  Jun-Wei Hsieh,et al.  Robust Rear Light Status Recognition Using Symmetrical SURFs , 2015, 2015 IEEE 18th International Conference on Intelligent Transportation Systems.

[15]  C. Tomasi.  Tracking Point Features , 2015 .

[16]  Cheng Xu,et al.  Vision-Based Method for Forward Vehicle Brake Lights Recognition , 2015 .

[17]  Senem Velipasalar,et al.  Robust and Computationally Lightweight Autonomous Tracking of Vehicle Taillights and Signal Detection by Embedded Smart Cameras , 2015, IEEE Transactions on Industrial Electronics.

[18]  Trevor Darrell,et al.  Long-term recurrent convolutional networks for visual recognition and description , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Zhiwei Song,et al.  Appearance-based Brake-Lights recognition using deep learning and vehicle detection , 2016, 2016 IEEE Intelligent Vehicles Symposium (IV).

[20]  Ming-Hsuan Yang,et al.  Learning to tell brake lights with convolutional features , 2016, 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC).

[21]  Hua-Tsung Chen,et al.  Daytime Preceding Vehicle Brake Light Detection Using Monocular Vision , 2016, IEEE Sensors Journal.

[22]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.