Fault Tolerant Perception for Urban Autonomous Vehicles

Road driving environments are complex, unstructured and highly changeable. A safe driving is, thus, becoming quite challenging task, in particular from the view point of development and deployment of autonomous vehicles-based urban transport systems. And, in that context, the reliable perception appears as a one of the main enabling strategies in developing safe autonomous driving. Currently, many autonomous vehicles are being tested on public roads with the objective of demonstrating the capability of operating in real world situations. A big effort has been focused towards creating fault-free autonomous vehicles. Nevertheless, fault tolerant perception for autonomous vehicles still needs to be further developed in order to create autonomous vehicles capable of driving under real road traffic conditions since on-board vehicle sensors may fail due to bad calibration, erroneous readings, physical or electrical failures, etc. A multi-sensor based vehicle architecture is a logical response to this issue. While the multi- sensor concept often relates to the strategy of using a variety of sensor types, this research has been focused on to the case when all sensors are vision sensors, either identical or different from each other. This thesis proposes a Fault Tolerant Perception paradigm that deals with possible sensor faults by defining the Federated Data Fusion Architecture designed to detect a faulty sensor and reduce its impact on to the safe autonomous driving. The proposed architecture minimises the influence of faulty data allowing the system to enter in a tolerated error state, where a recovery action can be performed to avoid failures. The developed architecture was then adapted towards meeting requirements of the KITTI Vision Benchmark Suite. Experimental results demonstrated the feasibility of the developed fault tolerant perception paradigm to successfully detect early faulty data from a singular sensor and to minimise the influence of that faulty sensor in the fusion process.

[1]  P. C. Jha,et al.  Reliability growth modeling and optimal release policy under fuzzy environment of an N-version programming system incorporating the effect of fault removal efficiency , 2007, Int. J. Autom. Comput..

[2]  Ephrahim Garcia,et al.  Team Cornell's Skynet: Robust perception and planning in an urban environment , 2008, J. Field Robotics.

[3]  A. Amditis,et al.  Detection of maneuvers using evidence theory , 2008, 2008 IEEE Intelligent Vehicles Symposium.

[4]  Teresa Escobet,et al.  Parameter estimation methods for fault detection and isolation LAAS-CNRS UPC , 2001 .

[5]  Luke Fletcher,et al.  The MIT - Cornell Collision and Why It Happened , 2009, The DARPA Urban Challenge.

[6]  M.D. Srinath,et al.  A Fault-Tolerant Multisensor Navigation System Design , 1987, IEEE Transactions on Aerospace and Electronic Systems.

[7]  P. J. Lawrence,et al.  Comparison of federated and centralized Kalman filters with fault detection considerations , 1994, Proceedings of 1994 IEEE Position, Location and Navigation Symposium - PLANS'94.

[8]  Dah-Jye Lee,et al.  Review of stereo vision algorithms and their suitability for resource-limited systems , 2013, Journal of Real-Time Image Processing.

[9]  Mica R. Endsley,et al.  Toward a Theory of Situation Awareness in Dynamic Systems , 1995, Hum. Factors.

[10]  Andrei Furda,et al.  An object-oriented design of a World Model for autonomous city vehicles , 2010, 2010 IEEE Intelligent Vehicles Symposium.

[11]  Stéphane Houzelle,et al.  Contribution to multisensor fusion formalization , 1994, Robotics Auton. Syst..

[12]  Morgan Quigley,et al.  ROS: an open-source Robot Operating System , 2009, ICRA 2009.

[13]  Padmanabhan Krishnan,et al.  Comparing model checking and static program analysis: A case study in error detection approaches , 2010 .

[14]  C. W. Chan,et al.  Application of Fully Decoupled Parity Equation in Fault Detection and Identification of DC Motors , 2006, IEEE Transactions on Industrial Electronics.

[15]  Erik Blasch,et al.  Level 5: user refinement to aid the fusion process , 2003, SPIE Defense + Commercial Sensing.

[16]  Angelos Amditis,et al.  Track based multi sensor data fusion for collision mitigation , 2008, 2008 11th International Conference on Information Fusion.

[17]  Saurabh Upadhyay,et al.  Optical Flow Measurement using Lucas Kanade Method , 2013 .

[18]  Alan N. Steinberg,et al.  Revisions to the JDL data fusion model , 1999, Defense, Security, and Sensing.

[19]  Thorsten Joachims,et al.  Making large-scale support vector machine learning practical , 1999 .

[20]  Li-Wei Fong,et al.  Multi‐sensor track‐to‐track fusion via linear minimum variance sense estimators , 2008 .

[21]  Carl E. Landwehr,et al.  Basic concepts and taxonomy of dependable and secure computing , 2004, IEEE Transactions on Dependable and Secure Computing.

[22]  Oddvar Hallingstad,et al.  Toward a framework for high integrity navigation of autonomous underwater vehicles , 2008 .

[23]  T Xu,et al.  A multi-sensor data fusion navigation system for an unmanned surface vehicle , 2007 .

[24]  Paul E. Rybski,et al.  Obstacle Detection and Tracking for the Urban Challenge , 2009, IEEE Transactions on Intelligent Transportation Systems.

[25]  M. Bedworth,et al.  The Omnibus model: a new model of data fusion? , 2000 .

[26]  Chingiz Hajiyev,et al.  Fault tolerant integrated radar/inertial altimeter based on Nonlinear Robust Adaptive Kalman filter , 2012 .

[27]  Marc Dacier A Fault Forecasting Approach for Operational Security Monitoring , 1995 .

[28]  Ivan Kadar Results from levels 2/3 fusion implementations: Issues, challenges, retrospectives and perspectives for the future - An annotated view , 2007, 2007 10th International Conference on Information Fusion.

[29]  Christoph Stiller,et al.  Velodyne SLAM , 2011, 2011 IEEE Intelligent Vehicles Symposium (IV).

[30]  James M. Hereford,et al.  Fault-tolerant sensor systems using evolvable hardware , 2006, IEEE Transactions on Instrumentation and Measurement.

[31]  Sunil Arya,et al.  An optimal algorithm for approximate nearest neighbor searching fixed dimensions , 1998, JACM.

[32]  M. Sivak,et al.  A PRELIMINARY ANALYSIS OF REAL-WORLD CRASHES INVOLVING SELF-DRIVING VEHICLES , 2015 .

[33]  Harold W. Kuhn,et al.  The Hungarian method for the assignment problem , 1955, 50 Years of Integer Programming.

[34]  Jannik Fritsch,et al.  A new performance measure and evaluation benchmark for road detection algorithms , 2013, 16th International IEEE Conference on Intelligent Transportation Systems (ITSC 2013).

[35]  Panlop Zeephongsekul,et al.  Recovery block fault tolerance considering correlated failures , 2014, 2014 Reliability and Maintainability Symposium.

[36]  Daniel D. Lee,et al.  Little Ben: The Ben Franklin Racing Team's entry in the 2007 DARPA Urban Challenge , 2008, J. Field Robotics.

[37]  Venkat Venkatasubramanian,et al.  Automatic generation of qualitative descriptions of process trends for fault detection and diagnosis , 1991 .

[38]  Peter King,et al.  Odin: Team VictorTango's entry in the DARPA Urban Challenge , 2008, J. Field Robotics.

[39]  Rolf Isermann,et al.  Fault-Diagnosis Applications: Model-Based Condition Monitoring: Actuators, Drives, Machinery, Plants, Sensors, and Fault-tolerant Systems , 2011 .

[40]  Miguel Realpe,et al.  Multi-sensor Fusion Module in a Fault Tolerant Perception System for Autonomous Vehicles , 2016 .

[41]  Rolf Isermann,et al.  COMPARISON OF CHANGE DETECTION METHODS FOR A RESIDUAL OF A HYDRAULIC SERVO-AXIS , 2005 .

[42]  Rong Wang,et al.  Research on scheme and algorithm of high-precision fault-tolerant integrated navigation for HALE UAV , 2010, 2010 3rd International Symposium on Systems and Control in Aeronautics and Astronautics.

[43]  E. R. Cohen An Introduction to Error Analysis: The Study of Uncertainties in Physical Measurements , 1998 .

[44]  Miguel Realpe,et al.  Towards fault tolerant perception for autonomous vehicles: Local fusion , 2015, 2015 IEEE 7th International Conference on Cybernetics and Intelligent Systems (CIS) and IEEE Conference on Robotics, Automation and Mechatronics (RAM).

[45]  Simon Lacroix,et al.  Calibration of a rotating multi-beam lidar , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[46]  Andreas Geiger,et al.  Vision meets robotics: The KITTI dataset , 2013, Int. J. Robotics Res..

[47]  James Llinas,et al.  Handbook of Multisensor Data Fusion , 2001 .

[48]  Luke Fletcher,et al.  A perception‐driven autonomous urban vehicle , 2008, J. Field Robotics.

[49]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[50]  G. Heredia,et al.  Sensor fault detection in small autonomous helicopters using observer/Kalman filter identification , 2009, 2009 IEEE International Conference on Mechatronics.

[51]  Angelos Amditis,et al.  Revisiting JDL model for automotive safety applications: the PF2 functional model , 2006, 2006 9th International Conference on Information Fusion.

[52]  Rubén Heras Evangelio,et al.  Clustering Motion for Real-Time Optical Flow Based Tracking , 2012, 2012 IEEE Ninth International Conference on Advanced Video and Signal-Based Surveillance.

[53]  Jeremy Broughton,et al.  Getting off your bike: cycling accidents in Great Britain in 1990-1999. , 2003, Accident; analysis and prevention.

[54]  William Whittaker,et al.  Autonomous driving in urban environments: Boss and the Urban Challenge , 2008, J. Field Robotics.

[55]  Kari Pulli,et al.  Real-time computer vision with OpenCV , 2012, Commun. ACM.

[56]  Christopher D. Wickens,et al.  Situation Awareness: Review of Mica Endsley's 1995 Articles on Situation Awareness Theory and Measurement , 2008, Hum. Factors.

[57]  Steven M. LaValle,et al.  Planning algorithms , 2006 .

[58]  Bernard Burg,et al.  An Approach to Data Fusion for Context Awareness , 2005, CONTEXT.

[59]  Sarfraz Khurshid,et al.  Directed incremental symbolic execution , 2011, PLDI '11.

[60]  Huadong Yu,et al.  Study on Fault-tolerant Filter Algorithm for Integrated Navigation System , 2007, 2007 International Conference on Mechatronics and Automation.

[61]  Karl-Erwin Großpietsch,et al.  Fault tolerance , 1994, IEEE Micro.

[62]  Radu Bogdan Rusu,et al.  Semantic 3D Object Maps for Everyday Manipulation in Human Living Environments , 2010, KI - Künstliche Intelligenz.

[63]  Sebastian Thrun,et al.  Junior: The Stanford entry in the Urban Challenge , 2008, J. Field Robotics.

[64]  Sameer Singh,et al.  Approaches to Multisensor Data Fusion in Target Tracking: A Survey , 2006, IEEE Transactions on Knowledge and Data Engineering.

[65]  Brian Randell,et al.  System structure for software fault tolerance , 1975, IEEE Transactions on Software Engineering.

[66]  Jitendra R. Raol,et al.  Multi-Sensor Data Fusion with MATLAB® , 2009 .

[67]  N. A. Carlson,et al.  Federated filter for fault-tolerant integrated navigation systems , 1988, IEEE PLANS '88.,Position Location and Navigation Symposium, Record. 'Navigation into the 21st Century'..

[68]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[69]  Paul Mineiro,et al.  Robust Sensor Fusion: Analysis and Application to Audio Visual Speech Recognition , 1998, Machine Learning.

[70]  David L. Hall,et al.  Dirty Secrets in Multisensor Data Fusion , 2001 .

[71]  Lawrence A. Klein,et al.  Sensor and Data Fusion: A Tool for Information Assessment and Decision Making , 2004 .

[72]  David L. Hall,et al.  Customer-Driven Sensor Management , 2006, IEEE Intelligent Systems.

[73]  H A Aldridge,et al.  Robot Position Sensor Fault Tolerance , 1997 .

[74]  Weigong Zhang,et al.  An Adaptive Fault-Tolerant Multisensor Navigation Strategy for Automated Vehicles , 2010, IEEE Transactions on Vehicular Technology.

[75]  Andreas Geiger,et al.  Are we ready for autonomous driving? The KITTI vision benchmark suite , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[76]  Toby P. Breckon,et al.  An empirical comparison of real-time dense stereo approaches for use in the automotive environment , 2012, EURASIP J. Image Video Process..

[77]  Raúl Rojas,et al.  Spirit of Berlin: An Autonomous Car for the DARPA Urban Challenge Hardware and Software Architecture , 2007 .

[78]  Arnaud Venet A practical approach to formal software verification by static analysis , 2008, ALET.

[79]  Kurt Konolige,et al.  Small Vision Systems: Hardware and Implementation , 1998 .

[80]  Angelos Amditis,et al.  Situation refinement for vehicle maneuver identification and driver's intention prediction , 2007, 2007 10th International Conference on Information Fusion.

[81]  Ljubo Vlacic,et al.  Sensor fault detection and diagnosis for autonomous vehicles , 2015 .

[82]  Andrei Furda,et al.  Enabling Safe Autonomous Driving in Real-World City Traffic Using Multiple Criteria Decision Making , 2011, IEEE Intelligent Transportation Systems Magazine.

[83]  José-Joel Gonzalez-Barbosa,et al.  LIDAR Velodyne HDL-64E Calibration Using Pattern Planes , 2011 .

[84]  SpeechRecognitionJavier R. Movellan Modularity and Catastrophic Fusion: a Bayesian Approach with Applications to Audiovisual Speech Recognition Modularity and Catastrophic Fusion: a Bayesian Approach with Applications to Audiovisual Speech Recognition , 1996 .

[85]  Keith Redmill,et al.  Systems for Safety and Autonomous Behavior in Cars: The DARPA Grand Challenge Experience , 2007, Proceedings of the IEEE.

[86]  Algirdas Avizienis,et al.  The N-Version Approach to Fault-Tolerant Software , 1985, IEEE Transactions on Software Engineering.

[87]  Ana Paula Kersting,et al.  A strip adjustment procedure to mitigate the impact of inaccurate mounting parameters in parallel lidar strips , 2009 .

[88]  Karl Nienhaus,et al.  Observer Based Condition Monitoring of the Generator Temperature Integrated in the Wind Turbine Controller , 2013 .

[89]  A. Edelmayer,et al.  Federated filtering for fault tolerant estimation and sensor redundancy management in coupled dynamics distributed systems , 2007, 2007 Mediterranean Conference on Control & Automation.