Deep CNN-Based Real-Time Traffic Light Detector for Self-Driving Vehicles

Due to the unavailability of Vehicle-to-Infrastructure (V2I) communication in current transportation systems, Traffic Light Detection (TLD) is still considered an important module in autonomous vehicles and Driver Assistance Systems (DAS). To overcome low flexibility and accuracy of vision-based heuristic algorithms and high power consumption of deep learning-based methods, we propose a lightweight and real-time traffic light detector for the autonomous vehicle platform. Our model consists of a heuristic candidate region selection module to identify all possible traffic lights, and a lightweight Convolution Neural Network (CNN) classifier to classify the results obtained. Offline simulations on the GPU server with the collected dataset and several public datasets show that our model achieves higher average accuracy and less time consumption. By integrating our detector module on NVidia Jetson TX1/TX2, we conduct on-road tests on two full-scale self-driving vehicle platforms (a car and a bus) in normal traffic conditions. Our model can achieve an average detection accuracy of 99.3 percent (mRttld) and 99.7 percent (Rttld) at 10Hz on TX1 and TX2, respectively. The on-road tests also show that our traffic light detection module can achieve <inline-formula><tex-math notation="LaTeX">$<\pm\; 1.5m$</tex-math><alternatives><mml:math><mml:mrow><mml:mo><</mml:mo><mml:mo>±</mml:mo><mml:mspace width="0.277778em"/><mml:mn>1</mml:mn><mml:mo>.</mml:mo><mml:mn>5</mml:mn><mml:mi>m</mml:mi></mml:mrow></mml:math><inline-graphic xlink:href="guizani-ieq1-2892451.gif"/></alternatives></inline-formula> errors at stop lines when working with other self-driving modules.

[1]  Huimin Ma,et al.  Traffic Light Recognition for Complex Scene With Fusion Detections , 2018, IEEE Transactions on Intelligent Transportation Systems.

[2]  Samy El-Tawab,et al.  Automatic Incident Detection in Intelligent Transportation Systems Using Aggregation of Traffic Parameters Collected Through V2I Communications , 2017, IEEE Intelligent Transportation Systems Magazine.

[3]  Kaiming He,et al.  Feature Pyramid Networks for Object Detection , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Xiaoping Du,et al.  Vision-Based Traffic Light Detection for Intelligent Vehicles , 2017, 2017 4th International Conference on Information Science and Control Engineering (ICISCE).

[5]  Ming Yang,et al.  Integrating visual selective attention model with HOG features for traffic light detection and recognition , 2015, 2015 IEEE Intelligent Vehicles Symposium (IV).

[6]  Chadi Assi,et al.  Multihop V2I Communications: A Feasibility Study, Modeling, and Performance Analysis , 2017, IEEE Transactions on Vehicular Technology.

[7]  Krishna Reddy Konda,et al.  An efficient vision-based traffic light detection and state recognition for autonomous vehicles , 2017, 2017 IEEE Intelligent Vehicles Symposium (IV).

[8]  Ju H. Park,et al.  Effective Traffic Lights Recognition Method for Real Time Driving Assistance Systemin the Daytime , 2011 .

[9]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Xinming Huang,et al.  Accurate and Reliable Detection of Traffic Lights Using Multiclass Learning and Multiobject Tracking , 2016, IEEE Intelligent Transportation Systems Magazine.

[11]  Alex Krizhevsky,et al.  One weird trick for parallelizing convolutional neural networks , 2014, ArXiv.

[12]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[13]  Evangeline Pollard,et al.  Tracking both pose and status of a traffic light via an Interacting Multiple Model filter , 2014, 17th International Conference on Information Fusion (FUSION).

[14]  Thomas B. Moeslund,et al.  Vision-Based Traffic Sign Detection and Analysis for Intelligent Driver Assistance Systems: Perspectives and Survey , 2012, IEEE Transactions on Intelligent Transportation Systems.

[15]  Qiang Chen,et al.  Network In Network , 2013, ICLR.

[16]  Mohan M. Trivedi,et al.  Traffic Light Detection: A Learning Algorithm and Evaluations on Challenging Dataset , 2015, 2015 IEEE 18th International Conference on Intelligent Transportation Systems.

[17]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[18]  Mohan M. Trivedi,et al.  Vision for Looking at Traffic Lights: Issues, Survey, and Perspectives , 2016, IEEE Transactions on Intelligent Transportation Systems.

[19]  Fernando Santos Osório,et al.  Traffic lights detection and state estimation using Hidden Markov Models , 2014, 2014 IEEE Intelligent Vehicles Symposium Proceedings.

[20]  Marcelo H. Ang,et al.  Traffic light status detection using movement patterns of vehicles , 2016, 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC).

[21]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[22]  Karsten Behrendt,et al.  A deep learning approach to traffic lights: Detection, tracking, and classification , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[23]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Mohan M. Trivedi,et al.  Traffic Light Detection at Night: Comparison of a Learning-Based Detector and Three Model-Based Detectors , 2015, ISVC.

[25]  Forrest N. Iandola,et al.  SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size , 2016, ArXiv.

[26]  Changshui Zhang,et al.  Real-Time Traffic Light Detection With Adaptive Background Suppression Filter , 2016, IEEE Transactions on Intelligent Transportation Systems.

[27]  Yu Liu,et al.  A cGANs-Based Scene Reconstruction Model Using Lidar Point Cloud , 2017, 2017 IEEE International Symposium on Parallel and Distributed Processing with Applications and 2017 IEEE International Conference on Ubiquitous Computing and Communications (ISPA/IUCC).

[28]  Senem Velipasalar,et al.  Mobile Standards-Based Traffic Light Detection in Assistive Devices for Individuals with Color-Vision Deficiency , 2015, IEEE Transactions on Intelligent Transportation Systems.

[29]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Johann Marius Zöllner,et al.  DeepTLR: A single deep convolutional network for detection and classification of traffic lights , 2016, 2016 IEEE Intelligent Vehicles Symposium (IV).

[31]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[33]  Jieyu Zhao,et al.  Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints , 2017, EMNLP.

[34]  Abdelwadood Mesleh,et al.  Traffic light detection for colorblind individuals , 2017, 2017 IEEE Jordan Conference on Applied Electrical Engineering and Computing Technologies (AEECT).

[35]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[36]  Yi Li,et al.  R-FCN: Object Detection via Region-based Fully Convolutional Networks , 2016, NIPS.

[37]  Pietro Cerri,et al.  Robust real-time traffic light detection and distance estimation using a single camera , 2015, Expert Syst. Appl..

[38]  Xinming Huang,et al.  Automatic detection of traffic lights using support vector machine , 2015, 2015 IEEE Intelligent Vehicles Symposium (IV).

[39]  Walid Saad,et al.  Machine Learning for Wireless Networks with Artificial Intelligence: A Tutorial on Neural Networks , 2017, ArXiv.

[40]  Zheng Liu,et al.  Saliency Map Generation by the Convolutional Neural Network for Real-Time Traffic Light Detection Using Template Matching , 2015, IEEE Transactions on Computational Imaging.