Automatic Sensor Correction of Autonomous Vehicles by Human-Vehicle Teaching-and-Learning

Autonomous driving technologies can provide greater safety, comfort, and efficiency for future transportation systems. Until now, much of the research effort has been devoted to developing different sensing and control algorithms. However, there has been limited research on how to handle sensor errors efficiently. A simple error in the sensor may lead to an unexpected failure in the whole autonomous driving function. In those cases, the vehicle is then recommended to be sent back to the manufacturer for repair, which costs time and money. This paper introduces an efficient automatic online sensor correction method. The method includes four major functions: sensor error detection, human teaching, vehicle learning, and vehicle self-evaluation. The first function is assumed to be ready and the major contribution of this paper is the human-vehicle teaching and learning framework, which utilizes human-vehicle interaction to collaboratively adjust the parameter in the control model in order to compensate for the errors of the sensors. The self-evaluation function is also briefly introduced. The applications of this method to radar and vision sensors to recover adaptive cruise control and lane keeping functions are introduced in detail. Experimental results acquired from high-fidelity 1/10-scale autonomous driving vehicles illustrate the effectiveness and advantages of the proposed approach.

[1]  D. Lichti SELF-CALIBRATION OF A 3 D RANGE CAMERA , 2008 .

[2]  Bart van Arem,et al.  The Impact of Cooperative Adaptive Cruise Control on Traffic-Flow Characteristics , 2006, IEEE Transactions on Intelligent Transportation Systems.

[3]  Robert Weigel,et al.  Employing six-port technology for phase-measurement-based calibration of automotive radar , 2009, 2009 Asia Pacific Microwave Conference.

[4]  Patrick Y. Shinzato,et al.  Fast visual road recognition and horizon detection using multiple artificial neural networks , 2012, 2012 IEEE Intelligent Vehicles Symposium.

[5]  Gaurav S. Sukhatme,et al.  Visual-Inertial Sensor Fusion: Localization, Mapping and Sensor-to-Sensor Self-calibration , 2011, Int. J. Robotics Res..

[6]  Sebastian Thrun,et al.  Towards fully autonomous driving: Systems and algorithms , 2011, 2011 IEEE Intelligent Vehicles Symposium (IV).

[7]  Giovanni Muscato,et al.  Localization and self-calibration of a robot for volcano exploration , 2004, IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004.

[8]  Christopher Edwards,et al.  Sliding mode observers for detection and reconstruction of sensor faults , 2002, Autom..

[9]  José-Joel Gonzalez-Barbosa,et al.  LIDAR Velodyne HDL-64E Calibration Using Pattern Planes , 2011 .

[10]  Lili Huang,et al.  A novel multi-planar LIDAR and computer vision calibration procedure using 2D patterns for automated navigation , 2009, 2009 IEEE Intelligent Vehicles Symposium.

[11]  Chee Pin Tan,et al.  Sliding mode observers for fault detection and isolation , 2002 .

[12]  Thomas F. Coleman,et al.  An Interior Trust Region Approach for Nonlinear Minimization Subject to Bounds , 1993, SIAM J. Optim..

[13]  Andreas Geiger,et al.  Automatic camera and range sensor calibration using a single shot , 2012, 2012 IEEE International Conference on Robotics and Automation.

[14]  Fadi Dornaika,et al.  Simultaneous robot-world and hand-eye calibration , 1998, IEEE Trans. Robotics Autom..

[15]  David M. Himmelblau,et al.  Sensor Fault Detection via Multiscale Analysis and Dynamic PCA , 1999 .

[16]  Shumeet Baluja,et al.  Evolution of an artificial neural network based autonomous land vehicle controller , 1996, IEEE Trans. Syst. Man Cybern. Part B.

[17]  E. Zafiriou,et al.  Use of neural networks for sensor failure detection in a control system , 1990, IEEE Control Systems Magazine.

[18]  Jafar Amiri Parian,et al.  Sensor modeling, self-calibration and accuracy testing of panoramic cameras and laser scanners , 2010 .

[19]  S Marsili-Libelli,et al.  Real-time fault detection and isolation in biological wastewater treatment plants. , 2009, Water science and technology : a journal of the International Association on Water Pollution Research.

[20]  M. Napolitano,et al.  Neural-network-based scheme for sensor failure detection, identification, and accommodation , 1995 .

[21]  Richard Bishop Intelligent Vehicle Applications Worldwide , 2000, IEEE Intell. Syst..

[22]  Patrick Rives,et al.  Single View Point Omnidirectional Camera Calibration from Planar Grids , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[23]  Yunyi Jia,et al.  Teaching Autonomous Vehicles How to Drive under Sensing Exceptions by Human Driving Demonstrations , 2017 .

[24]  Lennart Svensson,et al.  LIDAR-based driving path generation using fully convolutional neural networks , 2017, 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC).

[25]  Christopher Edwards,et al.  Sliding mode observers for robust detection and reconstruction of actuator and sensor faults , 2003 .

[26]  Kaouther Laabidi,et al.  Multimodel Approach Applied For Failure Diagnosis , 2008 .

[27]  William Whittaker,et al.  Autonomous driving in urban environments: Boss and the Urban Challenge , 2008, J. Field Robotics.

[28]  Peter S. Maybeck,et al.  Sensor/actuator failure detection in the Vista F-16 by multiple model adaptive estimation , 1995, IEEE Transactions on Aerospace and Electronic Systems.

[29]  Liqiang Feng,et al.  Measurement and correction of systematic odometry errors in mobile robots , 1996, IEEE Trans. Robotics Autom..

[30]  Xin Zhang,et al.  End to End Learning for Self-Driving Cars , 2016, ArXiv.

[31]  Robert Michael Lewis,et al.  Pattern Search Algorithms for Bound Constrained Minimization , 1999, SIAM J. Optim..

[32]  Paul Newman,et al.  Choosing a time and place for calibration of lidar-camera systems , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[33]  Jean Ponce,et al.  Computer Vision: A Modern Approach , 2002 .

[34]  Emanuele Menegatti,et al.  A robust and easy to implement method for IMU calibration without external equipments , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[35]  T.-H. Guo,et al.  Sensor failure detection and recovery by neural networks , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[36]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[37]  Dean Pomerleau,et al.  Efficient Training of Artificial Neural Networks for Autonomous Navigation , 1991, Neural Computation.

[38]  Roland Siegwart,et al.  Extrinsic self calibration of a camera and a 3D laser range finder from natural scenes , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[39]  Yunyi Jia,et al.  Determining Headway for Personalized Autonomous Vehicles by Learning from Human Driving Demonstration , 2017, 2017 IEEE 7th Annual International Conference on CYBER Technology in Automation, Control, and Intelligent Systems (CYBER).

[40]  Roland Siegwart,et al.  Theoretical Results on On-line Sensor Self-Calibration , 2006, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[41]  In-Beum Lee,et al.  Sensor fault identification based on time-lagged PCA in dynamic processes , 2004 .

[42]  Denis Fernando Wolf,et al.  A Road Following Approach Using Artificial Neural Networks Combinations , 2011, J. Intell. Robotic Syst..

[43]  H. Hartley The Modified Gauss-Newton Method for the Fitting of Non-Linear Regression Functions by Least Squares , 1961 .

[44]  J. J. Moré,et al.  Levenberg--Marquardt algorithm: implementation and theory , 1977 .

[45]  Mohan M. Trivedi,et al.  Video-based lane estimation and tracking for driver assistance: survey, system, and evaluation , 2006, IEEE Transactions on Intelligent Transportation Systems.

[46]  Andreas Geiger,et al.  Are we ready for autonomous driving? The KITTI vision benchmark suite , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[47]  Jin Jiang,et al.  An interacting multiple-model based fault detection, diagnosis and fault-tolerant control approach , 1999, Proceedings of the 38th IEEE Conference on Decision and Control (Cat. No.99CH36304).

[48]  Yann LeCun,et al.  Off-Road Obstacle Avoidance through End-to-End Learning , 2005, NIPS.

[49]  Sebastian Thrun,et al.  Model based vehicle detection and tracking for autonomous urban driving , 2009, Auton. Robots.