INCREMENTAL REAL-TIME BUNDLE ADJUSTMENT FOR MULTI-CAMERA SYSTEMS WITH POINTS AT INFINITY

This paper presents a concept and first experiments on a keyframe-based incremental bundle adjustment for real-time structure and motion estimation in an unknown scene. In order to avoid periodic batch steps, we use the software iSAM2 for sparse nonlinear incremental optimization, which is highly efficient through incremental variable reordering and fluid relinearization. We adapted the software to allow for (1) multi-view cameras by taking the rigid transformation between the cameras into account, (2) omnidirectional cameras as it can handle arbitrary bundles of rays and (3) scene points at infinity, which improve the estimation of the camera orientation as points at the horizon can be observed over long periods of time. The real-time bundle adjustment refers to sets of keyframes, consisting of frames, one per camera, taken in a synchronized way, that are initiated if a minimal geometric distance to the last keyframe set is exceeded. It uses interest points in the keyframes as observations, which are tracked in the synchronized video streams of the individual cameras and matched across the cameras, if possible. First experiments show the potential of the incremental bundle adjustment w.r.t. time requirements. Our experiments are based on a multi-camera system with four fisheye cameras, which are mounted on a UAV as two stereo pairs, one looking ahead and one looking backwards, providing a large field of view.

[1]  Hauke Strasdat,et al.  Visual SLAM: Why filter? , 2012, Image Vis. Comput..

[2]  Kurt Konolige,et al.  Visual Odometry Using Sparse Bundle Adjustment on an Autonomous Outdoor Vehicle , 2006, AMS.

[3]  Armin Gruen,et al.  Towards real-time photogrammetry , 1988 .

[4]  K. P. Schwarz,et al.  Digital image georeferencing from a multiple camera system by GPS/INS , 2001 .

[5]  Gene H. Golub,et al.  Matrix computations (3rd ed.) , 1996 .

[6]  Frank Dellaert,et al.  The Bayes Tree: An Algorithmic Foundation for Probabilistic Robot Mapping , 2010, WAFR.

[7]  Daniel G. Aliaga.  Accurate catadioptric calibration for real-time pose estimation in room-size environments , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[8]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[9]  Johannes Schneider,et al.  Bundle Adjustment and System Calibration with Points at Infinity for Omnidirectional Cameras , 2013 .

[10]  Andrew J. Davison,et al.  Real-time simultaneous localisation and mapping with a single camera , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[11]  A. Gruen.  Algorithmic aspects in on-line triangulation , 1985 .

[12]  Wolfgang Förstner,et al.  Bundle Adjustment and System Calibration with Points at Infinity for Omnidirectional Camera Systems Bündelausgleichung und Systemkalibrierung mit Punkten im Unendlichen für omnidirektionale Kamerasysteme , 2013 .

[13]  Kostas Daniilidis,et al.  Monocular visual odometry in urban environments using an omnidirectional camera , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[14]  Wolfgang Förstner,et al.  Fish-Eye-Stereo Calibration and Epipolar Rectification , 2005 .

[15]  F. Dellaert.  Factor Graphs and GTSAM: A Hands-on Introduction , 2012 .

[16]  Frank Dellaert,et al.  iSAM2: Incremental smoothing and mapping using the Bayes tree , 2012, Int. J. Robotics Res..

[17]  G. Klein,et al.  Parallel Tracking and Mapping for Small AR Workspaces , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[18]  Michel Dhome,et al.  Generic and real-time structure from motion using local bundle adjustment , 2009, Image Vis. Comput..

[19]  J.-Y. Bouguet,et al.  Pyramidal implementation of the lucas kanade feature tracker , 1999 .

[20]  Hans-Gerd Maas.  Image sequence based automatic multi-camera system calibration techniques 1 Revised version of a pap , 1999 .

[21]  Wolfgang Förstner,et al.  Bundle Adjustment for Multi-Camera Systems with Points at Infinity , 2012 .

[22]  Frank Dellaert,et al.  iSAM: Incremental Smoothing and Mapping , 2008, IEEE Transactions on Robotics.

[23]  Impyeong Lee,et al.  A Sequential Aerial Triangulation Algorithm for Real-time Georeferencing of Image Sequences Acquired by an Airborne Multi-Sensor System , 2012, Remote. Sens..