Brain-Inspired Cognitive Model With Attention for Self-Driving Cars

The perception-driven approach and end-to-end system are two major vision-based frameworks for self-driving cars. However, it is difficult to introduce attention and historical information into the autonomous driving process, which are essential for achieving human-like driving in these two methods. In this paper, we propose a novel model for self-driving cars called the brain-inspired cognitive model with attention. This model comprises three parts: 1) a convolutional neural network for simulating the human visual cortex; 2) a cognitive map to describe the relationships between objects in a complex traffic scene; and 3) a recurrent neural network, which is combined with the real-time updated cognitive map to implement the attention mechanism and long-short term memory. An advantage of our model is that it can accurately solve three tasks simultaneously: 1) detecting the free space and boundaries for the current and adjacent lanes; 2) estimating the distances to obstacles and vehicle attitude; and 3) learning the driving behavior and decision-making process of a human driver. Importantly, the proposed model can accept external navigation instructions during an end-to-end driving process. To evaluate the model, we built a large-scale road-vehicle dataset containing over 40 000 labeled road images captured by three cameras placed on our self-driving car. Moreover, human driving activities and vehicle states were recorded at the same time.

[1]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[2]  J. Hawkins,et al.  On Intelligence , 2004 .

[3]  Li Qing.  Calibration of external parameters of vehicle-mounted camera with trilinear method , 2004 .

[4]  Haizhou Li,et al.  An Entorhinal-Hippocampal Model for Simultaneous Cognitive Map Building , 2015, AAAI.

[5]  V. Mountcastle,et al.  An organizing principle for cerebral function : the unit module and the distributed system , 1978 .

[6]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Matthew Turk,et al.  VITS-A Vision System for Autonomous Land Vehicle Navigation , 1988, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  Yang Gao,et al.  End-to-End Learning of Driving Models from Large-Scale Video Datasets , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Jitendra Malik,et al.  Region-Based Convolutional Networks for Accurate Object Detection and Segmentation , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[11]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[12]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[13]  Emilio Frazzoli,et al.  A Survey of Motion Planning and Control Techniques for Self-Driving Urban Vehicles , 2016, IEEE Transactions on Intelligent Vehicles.

[14]  Asif Iqbal,et al.  Multiple lane boundary detection using a combination of low-level image features , 2014, 17th International IEEE Conference on Intelligent Transportation Systems (ITSC).

[15]  刘 子熠,et al.  Hybrid-augmented intelligence: collaboration and cognition , 2017, Frontiers of Information Technology & Electronic Engineering.

[16]  Haizhou Li,et al.  How the Brain Formulates Memory: A Spatio-Temporal Model Research Frontier , 2016, IEEE Computational Intelligence Magazine.

[17]  Sadayuki Tsugawa,et al.  Vision-based vehicles in Japan: machine vision systems and driving control systems , 1994, IEEE Trans. Ind. Electron..

[18]  Luke Fletcher,et al.  A perception‐driven autonomous urban vehicle , 2008, J. Field Robotics.

[19]  Jannik Fritsch,et al.  A new performance measure and evaluation benchmark for road detection algorithms , 2013, 16th International IEEE Conference on Intelligent Transportation Systems (ITSC 2013).

[20]  S. Khan,et al.  Real time lane detection for autonomous vehicles , 2008, 2008 International Conference on Computer and Communication Engineering.

[21]  Nanning Zheng,et al.  A vision-centered multi-sensor fusing approach to self-localization and obstacle perception for robotic cars , 2017, Frontiers of Information Technology & Electronic Engineering.

[22]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[23]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[24]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[25]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[26]  E. Tolman.  Cognitive maps in rats and men. , 1948, Psychological review.

[27]  Jürgen Schmidhuber,et al.  Evolving large-scale neural networks for vision-based TORCS , 2013, FDG.

[28]  Xin Zhang,et al.  End to End Learning for Self-Driving Cars , 2016, ArXiv.

[29]  Bruce L. McNaughton,et al.  Path integration and the neural basis of the 'cognitive map' , 2006, Nature Reviews Neuroscience.

[30]  Nanning Zheng,et al.  Efficient Lane Boundary Detection with Spatial-Temporal Knowledge Filtering , 2016, Sensors.

[31]  Jürgen Schmidhuber,et al.  Evolving large-scale neural networks for vision-based reinforcement learning , 2013, GECCO '13.

[32]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[33]  Fernando A. Mujica,et al.  An Empirical Evaluation of Deep Learning on Highway Driving , 2015, ArXiv.

[34]  Dean Pomerleau,et al.  ALVINN, an autonomous land vehicle in a neural network , 2015 .

[35]  Yann LeCun,et al.  Off-Road Obstacle Avoidance through End-to-End Learning , 2005, NIPS.

[36]  R. Passingham.  The hippocampus as a cognitive map J. O'Keefe & L. Nadel, Oxford University Press, Oxford (1978). 570 pp., £25.00 , 1979, Neuroscience.

[37]  Wei Huang,et al.  A Lane Detection Method for Lane Departure Warning System , 2010, 2010 International Conference on Optoelectronics and Image Processing.

[38]  Dinggang Shen,et al.  Lane detection and tracking using B-Snake , 2004, Image Vis. Comput..

[39]  Jianxiong Xiao,et al.  DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).