Applying robust structure from motion to markerless augmented reality

We demonstrate a complete system for markerless augmented reality using robust structure from motion. The proposed system includes two main components. The first is a means of learning the appearance of complex 3D objects and augmenting them with virtual annotations. Its output is database of recognizable landmarks along with 3D descriptions of accompanying virtual objects. The second component uses this data to recognize the previously learned landmarks, recover camera pose, and render the associated virtual content. Both components make use of the recently developed subtrack optimization algorithm for structure from motion, which we demonstrate to be a useful tool for both learning the structure of objects and tracking camera pose after recognition. The complete system is demonstrated on several complex real-world examples.

[1]  Vincent Lepetit,et al.  Multiple 3D Object tracking for augmented reality , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[2]  Tim Hauke Heibel,et al.  A mobile markerless AR system for maintenance and repair , 2006, 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality.

[3]  J.-Y. Bouguet,et al.  Pyramidal implementation of the lucas kanade feature tracker , 1999 .

[4]  Jeremiah Neubert,et al.  Semi-Autonomous Generation of Appearance-based Edge Models from Image Sequences , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[5]  Suya You,et al.  A Dynamic Programming Approach to Maximizing Tracks for Structure from Motion , 2009, ACCV.

[6]  Naokazu Yokoya,et al.  Real-time geometric registration using feature landmark database for augmented reality applications , 2009, Electronic Imaging.

[7]  Yacov Hel-Or,et al.  Real-Time Pattern Matching Using Projection Kernels , 2005, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  Didier Stricker,et al.  Online camera pose estimation in partially known and dynamic scenes , 2006, 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality.

[9]  Quan Wang,et al.  Real-Time Image Matching Based on Multiple View Kernel Projection , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[11]  G. Klein,et al.  Parallel Tracking and Mapping for Small AR Workspaces , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[12]  Suya You,et al.  Fast Simultaneous Tracking and Recognition Using Incremental Keypoint Matching , 2008 .

[13]  David G. Lowe,et al.  Shape indexing using approximate nearest-neighbour search in high-dimensional spaces , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[14]  Supun Samarasekera,et al.  Real-time global localization with a pre-built visual landmark database , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Volker Coors,et al.  Matching buildings: pose estimation in an urban environment , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[16]  Andrew W. Fitzgibbon,et al.  Markerless tracking using planar structures in the scene , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[17]  Nassir Navab,et al.  How to augment the second image? Recovery of the translation scale in image to image registration , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[18]  Vincent Lepetit,et al.  Randomized trees for real-time keypoint recognition , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[19]  Tom Drummond,et al.  Semi-automatic Annotations in Unknown Environments , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[20]  James J. Little,et al.  Vision-based global localization and mapping for mobile robots , 2005, IEEE Transactions on Robotics.

[21]  David G. Lowe,et al.  Scene modelling, recognition and tracking with invariant image features , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.