Detection of moving objects by spatio-temporal motion analysis

Driver assistance systems of the future, that will support the driver in complex driving situations, require a thorough understanding of the car's environment. This includes not only the comprehension of the infrastructure, but also the precise detection and measurement of other moving traffic participants. In this thesis, a novel principle is presented and investigated in detail, that allows the reconstruction of the 3d motion field from the image sequence obtained by a stereo camera system. Given correspondences of stereo measurements over time, this principle estimates the 3d position and the 3d motion vector of selected points using Kalman Filters, resulting in a real-time estimation of the observed motion field. Since the state vector of the Kalman Filter consists of six elements, this principle is called 6d-Vision. To estimate the absolute motion field, the ego-motion of the moving observer must be known precisely. Since cars are usually not equipped with high-end inertial sensors, a novel algorithm to estimate the ego-motion from the image sequence is presented. Based on a Kalman Filter, it is able to support even complex vehicle models, and takes advantage of all available data, namely the previously estimated motion field and eventually available inertial sensors. As the 6d-Vision principle is not restricted to particular algorithms to obtain the image measurements, various optical flow and stereo algorithms are evaluated. In particular, a novel dense stereo algorithm is presented, that gives excellent precision results and runs at real-time. In addition, two novel scene flow algorithms are introduced, that measure the optical flow and stereo information in a combined approach, yielding more precise and robust results than a separate analysis of the two information sources. The application of the 6d-Vision principle to real-world data is illustrated throughout the thesis. As practical applications usually require an object understanding, rather than a 3d motion field, a simple, yet efficient algorithm to detect and track moving objects is presented. This algorithm was successfully implemented in a demonstrator vehicle, that performs an autonomous braking resp. steering manoeuvre to avoid collisions with moving pedestrians.

[1]  Larry H. Matthies,et al.  Error modeling in stereo navigation , 1986, IEEE J. Robotics Autom..

[2]  O. Faugeras.  Three-dimensional computer vision: a geometric viewpoint , 1993 .

[3]  T. Brox,et al.  Universität Des Saarlandes Fachrichtung 6.1 – Mathematik a Survey on Variational Optic Flow Methods for Small Displacements a Survey on Variational Optic Flow Methods for Small Displacements , 2022 .

[4]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[5]  H. W. Sorenson,et al.  Kalman filtering : theory and application , 1985 .

[6]  Paul J. Besl,et al.  Method for registration of 3-D shapes , 1992, Other Conferences.

[7]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Greg Welch,et al.  Welch & Bishop , An Introduction to the Kalman Filter 2 1 The Discrete Kalman Filter In 1960 , 1994 .

[9]  C. Rabe,et al.  Kalman filter based depth from motion with fast convergence , 2005, IEEE Proceedings. Intelligent Vehicles Symposium, 2005..

[10]  Uwe Franke,et al.  Real-time stereo vision for urban traffic scene understanding , 2000, Proceedings of the IEEE Intelligent Vehicles Symposium 2000 (Cat. No.00TH8511).

[11]  Wolfgang Förstner,et al.  Simultaneous Estimation of Pose and Motion at Highly Dynamic Turn Maneuvers , 2009, DAGM-Symposium.

[12]  Wei-Song Lin,et al.  A 3D Predictive Visual Tracker for Tracking Multiple Moving Objects with a Stereo Vision System , 1995, ICSC.

[13]  J. Morat,et al.  Tracking with Stereo-vision System for Low Speed Following Applications , 2007, 2007 IEEE Intelligent Vehicles Symposium.

[14]  Daniel Cremers,et al.  Efficient Dense Scene Flow from Sparse or Dense Stereo Data , 2008, ECCV.

[15]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[16]  Uwe Franke,et al.  Kollisionsvermeidung durch raum-zeitliche Bildanalyse (Collision Avoidance based on Space-Time Image Analysis) , 2007, it Inf. Technol..

[17]  Christoph Stiller,et al.  Fusing optical flow and stereo disparity for object tracking , 2002, Proceedings. The IEEE 5th International Conference on Intelligent Transportation Systems.

[18]  Kenneth Levenberg.  A METHOD FOR THE SOLUTION OF CERTAIN NON – LINEAR PROBLEMS IN LEAST SQUARES , 1944 .

[19]  S. Shankar Sastry,et al.  An Invitation to 3-D Vision , 2004 .

[20]  Joris De Schutter,et al.  Kalman filters for nonlinear systems , 2002 .

[21]  James Llinas,et al.  Handbook of Multisensor Data Fusion , 2001 .

[22]  Richard Szeliski,et al.  A Database and Evaluation Methodology for Optical Flow , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[23]  David J. Fleet,et al.  Computation of component image velocity from local phase information , 1990, International Journal of Computer Vision.

[24]  Hans P. Moravec,et al.  Robot Evidence Grids. , 1996 .

[25]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[26]  A. Murat Tekalp,et al.  Simultaneous motion-disparity estimation and segmentation from stereo , 1994, Proceedings of 1st International Conference on Image Processing.

[27]  J. Klappstein,et al.  Monocular Motion Detection Using Spatial Constraints in a Unified Manner , 2006, 2006 IEEE Intelligent Vehicles Symposium.

[28]  Hans-Hellmut Nagel,et al.  An Investigation of Smoothness Constraints for the Estimation of Displacement Vector Fields from Image Sequences , 1983, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[29]  Hernán Badino,et al.  A Robust Approach for Ego-Motion Estimation Using a Mobile Stereo Platform , 2004, IWCM.

[30]  T. Vaudrey,et al.  Stereo-based Free Space Computation in Complex Traffic Scenarios , 2008, 2008 IEEE Southwest Symposium on Image Analysis and Interpretation.

[31]  Simon Lacroix,et al.  High resolution terrain mapping using low attitude aerial stereo imagery , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[32]  Emanuele Trucco,et al.  Introductory techniques for 3-D computer vision , 1998 .

[33]  Olivier D. Faugeras,et al.  The geometry of multiple images - the laws that govern the formation of multiple images of a scene and some of their applications , 2001 .

[34]  P J Kellman,et al.  Extracting object motion during observer motion: combining constraints from optic flow and binocular disparity. , 1995, Journal of the Optical Society of America. A, Optics, image science, and vision.

[35]  Uwe Franke,et al.  6D-Vision: Fusion of Stereo and Motion for Robust Environment Perception , 2005, DAGM-Symposium.

[36]  Trevor Darrell,et al.  Stereo tracking using ICP and normal flow constraint , 2002, Object recognition supported by user interaction for service robots.

[37]  Alain Crouzil,et al.  Dense matching using correlation: new measures that are robust near occlusions , 2003, BMVC.

[38]  D. Cremers,et al.  Duality TV-L1 flow with fundamental matrix prior , 2008, 2008 23rd International Conference Image and Vision Computing New Zealand.

[39]  Reg G. Willson.  Modeling and calibration of automated zoom lenses , 1994, Other Conferences.

[40]  Reinhard Koch,et al.  A simple and efficient rectification method for general motion , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[41]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[42]  Patrick Rives,et al.  Recursive Estimation of 3D Features Using Optical Flow and Camera Motion , 1986, Annual Meeting of the IEEE Industry Applications Society.

[43]  Boguslaw Cyganek.  Road-Signs Recognition System for Intelligent Vehicles , 2008, RobVis.

[44]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[45]  T. Clarke,et al.  The Development of Camera Calibration Methods and Models , 1998 .

[46]  C. Rabe,et al.  Fast detection of moving objects in complex scenarios , 2007, 2007 IEEE Intelligent Vehicles Symposium.

[47]  M. Shimizu,et al.  Precise sub-pixel estimation on area-based matching , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[48]  A. Murat Tekalp,et al.  Simultaneous stereo-motion fusion and 3-D motion tracking , 1995, 1995 International Conference on Acoustics, Speech, and Signal Processing.

[49]  David J. Fleet,et al.  Performance of optical flow techniques , 1994, International Journal of Computer Vision.

[50]  M. Pollefeys.  Self-calibration and metric 3d reconstruction from uncalibrated image sequences , 1999 .

[51]  Jean Ponce,et al.  Computer Vision: A Modern Approach , 2002 .

[52]  Radu Horaud,et al.  Motion-Egomotion Discrimination and Motion Segmentation from Image-Pair Streams , 2000, Comput. Vis. Image Underst..

[53]  Y. Hung,et al.  A 3D Feature-Based Tracker for Multiple Object Tracking , 1999 .

[54]  Larry H. Matthies,et al.  Kalman filter-based algorithms for estimating depth from image sequences , 1989, International Journal of Computer Vision.

[55]  Jean-Yves Bouguet,et al.  Camera calibration toolbox for matlab , 2001 .

[56]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[57]  Vladimir Kolmogorov,et al.  An experimental comparison of min-cut/max- flow algorithms for energy minimization in vision , 2001, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[58]  Alexander Barth,et al.  DYNAMIC STEREO VISION FOR INTERSECTION ASSISTANCE , 2008 .

[59]  Frank Suhling,et al.  Robust Obstacle Detection from Stereoscopic Image Sequences Using Kalman Filtering , 2001, DAGM-Symposium.

[60]  Y. Kay,et al.  A Kalman filter approach for accurate 3D motion estimation from a sequence of stereo images , 1990, [1990] Proceedings. 10th International Conference on Pattern Recognition.

[61]  Steven A. Shafer,et al.  Perspective projection camera model for zoom lenses , 1994, Other Conferences.

[62]  L. C. Jain,et al.  Advances in Intelligent Systems for Defence , 2002 .

[63]  Fridtjof Stein,et al.  In-factory calibration of multiocular camera systems , 2004, SPIE Photonics Europe.

[64]  Ting-Chuen Pong,et al.  Cooperative fusion of stereo and motion , 1994, Proceedings of ICSIPNN '94. International Conference on Speech, Image Processing and Neural Networks.

[65]  Jan A Snyman,et al.  Practical Mathematical Optimization: An Introduction to Basic Optimization Theory and Classical and New Gradient-Based Algorithms , 2005 .

[66]  Roger Y. Tsai,et al.  A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses , 1987, IEEE J. Robotics Autom..

[67]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[68]  John R. Schott,et al.  Remote Sensing: The Image Chain Approach , 1996 .

[69]  Fridtjof Stein,et al.  Efficient Computation of Optical Flow Using the Census Transform , 2004, DAGM-Symposium.

[70]  Qi Tian,et al.  Algorithms for subpixel registration , 1986 .

[71]  Alexandru Tupan,et al.  Triangulation , 1997, Comput. Vis. Image Underst..

[72]  Janne Heikkilä,et al.  A four-step camera calibration procedure with implicit image correction , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[73]  Jin Liu,et al.  Stereo and motion correspondence in a sequence of stereo images , 1993, Signal Process. Image Commun..

[74]  Rama Chellappa,et al.  Dynamic feature point tracking in an image sequence , 1994, Proceedings of 12th International Conference on Pattern Recognition.

[75]  Stefan K. Gehrig,et al.  A Real-Time Low-Power Stereo Vision Engine Using Semi-Global Matching , 2009, ICVS.

[76]  Antonis A. Argyros,et al.  Robust Regression for the Detection of Independent 3D Motion by a Binocular Observer , 1998, Real Time Imaging.

[77]  Ramin Zabih,et al.  Non-parametric Local Transforms for Computing Visual Correspondence , 1994, ECCV.

[78]  R Risack.  ROBUST LANE RECOGNITION EMBEDDED IN A REAL-TIME DRIVER ASSISTANCE SYSTEM , 1998 .

[79]  H. Hirschmüller.  Accurate and Efficient Stereo Processing by Semi-Global Matching and Mutual Information , 2005, CVPR.

[80]  Gaurav S. Sukhatme,et al.  Bias Reduction and Filter Convergence for Long Range Stereo , 2005, ISRR.

[81]  F. Lindner,et al.  Robust recognition of traffic signals , 2004, IEEE Intelligent Vehicles Symposium, 2004.

[82]  David J. Fleet,et al.  Optical Flow Estimation , 2006, Handbook of Mathematical Models in Computer Vision.

[83]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[84]  G. Bierman.  Factorization methods for discrete sequential estimation , 1977 .

[85]  Steven S. Beauchemin,et al.  The computation of optical flow , 1995, CSUR.

[86]  S. Heinrich.  Fast obstacle detection using flow/depth constraint , 2002, Intelligent Vehicle Symposium, 2002. IEEE.

[87]  Luke Fletcher,et al.  Real-Time Speed Sign Detection Using the Radial Symmetry Detector , 2008, IEEE Transactions on Intelligent Transportation Systems.

[88]  Takeo Kanade,et al.  Three-dimensional scene flow , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[89]  A. Jazwinski.  Stochastic Processes and Filtering Theory , 1970 .

[90]  Dean Brown,et al.  Decentering distortion of lenses , 1966 .

[91]  Larry H. Matthies,et al.  Real-time detection of moving objects from moving vehicles using dense stereo and optical flow , 2004, 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566).

[92]  T. Brox,et al.  Variational Optic Flow Computation: From Continuous Models to Algorithms , 2003 .

[93]  Clemens Rabe,et al.  6D Vision Goes Fisheye for Intersection Assistance , 2008, 2008 Canadian Conference on Computer and Robot Vision.

[94]  John K. Tsotsos,et al.  Applying temporal constraints to the dynamic stereo problem , 1986, Comput. Vis. Graph. Image Process..

[95]  Larry H. Matthies,et al.  Attenuating stereo pixel-locking via affine window adaptation , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[96]  Horst Bischof,et al.  A Duality Based Approach for Realtime TV-L1 Optical Flow , 2007, DAGM-Symposium.

[97]  Trevor Darrell,et al.  Motion Estimation from Disparity Images , 2001, ICCV.

[98]  Reinhard Klette,et al.  Moving Object Segmentation Using Optical Flow and Depth Information , 2009, PSIVT.

[99]  Kurt Konolige,et al.  Real-Time Detection of Independent Motion using Stereo , 2005, 2005 Seventh IEEE Workshops on Applications of Computer Vision (WACV/MOTION'05) - Volume 1.

[100]  Mordecai Avriel,et al.  Nonlinear programming , 1976 .

[101]  Uwe Franke,et al.  Kalman Filter based Detection of Obstacles and Lane Boundary in Monocular Image Sequences , 2005, AMS.

[102]  Hernán Badino.  Binocular ego-motion estimation for automotive applications , 2008 .

[103]  Uwe Franke,et al.  Towards Optimal Stereo Analysis of Image Sequences , 2008, RobVis.

[104]  T. Vaudrey,et al.  Evaluation of moving object segmentation comparing 6D-vision and monocular motion constraints , 2008, 2008 23rd International Conference Image and Vision Computing New Zealand.

[105]  Allen M. Waxman,et al.  Binocular Image Flows: Steps Toward Stereo-Motion Fusion , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[106]  A. Verri,et al.  A compact algorithm for rectification of stereo pairs , 2000 .