Multi-modal object detection and localization for high integrity driving assistance

Much work is currently devoted to increasing the reliability, completeness and precision of the data used by driving assistance systems, particularly in urban environments. Urban environments represent a particular challenge for the task of perception, since they are complex, dynamic and completely variable. This article examines a multi-modal perception approach for enhancing vehicle localization and the tracking of dynamic objects in a world-centric map. 3D ego-localization is achieved by merging stereo vision perception data and proprioceptive information from vehicle sensors. Mobile objects are detected using a multi-layer lidar that is simultaneously used to identify a zone of interest to reduce the complexity of the perception process. Object localization and tracking is then performed in a fixed frame which simplifies analysis and understanding of the scene. Finally, tracked objects are confirmed by vision using 3D dense reconstruction in focused regions of interest. Only confirmed objects can generate an alarm or an action on the vehicle. This is crucial to reduce false alarms that affect the trust that the driver places in the driving assistance system. Synchronization issues between the sensing modalities are solved using predictive filtering. Real experimental results are reported so that the performance of the multi-modal system may be evaluated.

[1]  Mohinder S. Grewal,et al.  Kalman Filtering: Theory and Practice , 1993 .

[2]  Véronique Cherfaoui,et al.  Object-level fusion and confidence management in a multi-sensor pedestrian tracking system , 2008 .

[3]  Patrick Rives,et al.  Accurate Quadrifocal Tracking for Robust 3D Visual Odometry , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[4]  Robert Pless,et al.  Extrinsic calibration of a camera and laser range finder (improves camera calibration) , 2004, 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566).

[5]  Steven D. Blostein,et al.  Correction to "Error Analysis in Stereo Determination of 3-D Point Positions" , 1988, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Philippe Fuchs,et al.  An improved calibration technique for coupled single-row telemeter and CCD camera , 2005, Fifth International Conference on 3-D Digital Imaging and Modeling (3DIM'05).

[7]  Andrew J. Hanson,et al.  Visualizing quaternions , 2005, SIGGRAPH Courses.

[8]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[9]  Roland Siegwart,et al.  Extrinsic self calibration of a camera and a 3D laser range finder from natural scenes , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[10]  S. Thrun,et al.  Model Based Vehicle Tracking in Urban Environments , 2009 .

[11]  Andrew Zisserman,et al.  Multiple View Geometry in Computer Vision (2nd ed) , 2003 .

[12]  Michael R. M. Jenkin,et al.  Inertial Sensors, GPS, and Odometry , 2008, Springer Handbook of Robotics.

[13]  Andrew J. Hanson Visualizing Quaternions (The Morgan Kaufmann Series in Interactive 3D Technology) , 2006 .

[14]  V. Cherfaoui,et al.  Tracking objects using a laser scanner in driving situation based on modeling target shape , 2007, 2007 IEEE Intelligent Vehicles Symposium.

[15]  Charles V. Stewart,et al.  Robust Parameter Estimation in Computer Vision , 1999, SIAM Rev..

[16]  Yaakov Bar-Shalom,et al.  Multitarget-Multisensor Tracking: Applications and Advances , 1992 .

[17]  Steven D. Blostein,et al.  Error Analysis in Stereo Determination of 3-D Point Positions , 1987, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  S. Thrun,et al.  Simultaneous Localization, Mapping and Moving Object Tracking Simultaneous Localization, Mapping and Moving Object Tracking , 2007 .

[19]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[20]  Ho Gi Jung,et al.  A New Approach to Urban Pedestrian Detection for Automatic Braking , 2009, IEEE Transactions on Intelligent Transportation Systems.

[21]  H. Durrant-Whyte,et al.  Simultaneous Localisation and Mapping ( SLAM ) : Part II State of the Art , 2006 .

[22]  Vincent Frémont,et al.  Extrinsic calibration between a multi-layer lidar and a camera , 2008, 2008 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems.

[23]  J.-Y. Bouguet,et al.  Pyramidal implementation of the lucas kanade feature tracker , 1999 .

[24]  Luc Van Gool,et al.  Dynamic 3D Scene Analysis from a Moving Vehicle , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[25]  Philippe Bonnifait,et al.  An experiment of a 3D real-time robust visual odometry for intelligent vehicles , 2009, 2009 12th International IEEE Conference on Intelligent Transportation Systems.

[26]  Dominique Gruyer,et al.  Cooperative Fusion for Multi-Obstacles Detection With Use of Stereovision and Laser Scanner , 2005, Auton. Robots.

[27]  Takeo Miyasaka,et al.  Ego-motion estimation and moving object tracking using multi-layer LIDAR , 2009, 2009 IEEE Intelligent Vehicles Symposium.

[28]  Michael Werman,et al.  Trilinearity of three perspective views and its associated tensor , 1995, Proceedings of IEEE International Conference on Computer Vision.

[29]  Hugh Durrant-Whyte,et al.  Simultaneous localization and mapping (SLAM): part II , 2006 .

[30]  J. MacQueen Some methods for classification and analysis of multivariate observations , 1967 .

[31]  Cindy Cappelle,et al.  Multi-sensors data fusion using Dynamic Bayesian Network for robotised vehicle geo-localisation , 2008, 2008 11th International Conference on Information Fusion.

[32]  Mohinder S. Grewal,et al.  Kalman Filtering: Theory and Practice Using MATLAB , 2001 .

[33]  James Llinas,et al.  Handbook of Multisensor Data Fusion : Theory and Practice, Second Edition , 2008 .

[34]  James Llinas,et al.  Multisensor Data Fusion , 1990 .

[35]  Christopher G. Harris,et al.  A Combined Corner and Edge Detector , 1988, Alvey Vision Conference.

[36]  Hugh F. Durrant-Whyte,et al.  Multisensor Data Fusion , 2008, Springer Handbook of Robotics, 2nd Ed..

[37]  Hugh F. Durrant-Whyte,et al.  Simultaneous localization and mapping: part I , 2006, IEEE Robotics & Automation Magazine.

[38]  Fawzi Nashashibi,et al.  Vehicle recognition and tracking using a generic multisensor and multialgorithm fusion approach , 2008 .