Real-time interactive modeling and scalable multiple object tracking for AR

We propose a real-time solution for modeling and tracking multiple 3D objects in unknown environments for Augmented Reality. The proposed solution consists of both scalable tracking and interactive modeling. Our contribution is twofold: First, we show how to scale with the number of objects using keyframes. This is done by combining recent techniques for image retrieval and online Structure from Motion, which can be run in parallel. As a result, tracking 50 objects in 3D can be done within 6-35ms per frame, even under difficult conditions for tracking. Second, we propose a method to let the user add new objects very quickly. The user simply has to select in an image a 2D region lying on the object. A 3D primitive is then fitted to the features within this region, and adjusted to create the object 3D model. We demonstrate the modeling of polygonal and circular-based objects. In practice, this procedure takes less than a minute.

[1]  Anton van den Hengel,et al.  In situ image-based modeling , 2009, 2009 8th IEEE International Symposium on Mixed and Augmented Reality.

[2]  Tom Drummond,et al.  Interactive model reconstruction with user guidance , 2009, 2009 8th IEEE International Symposium on Mixed and Augmented Reality.

[3]  Vincent Lepetit,et al.  Multiple 3D Object tracking for augmented reality , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[4]  Dieter Schmalstieg,et al.  Multiple target detection and tracking with guaranteed framerates on mobile phones , 2009, 2009 8th IEEE International Symposium on Mixed and Augmented Reality.

[5]  Vincent Lepetit,et al.  Extended Keyframe Detection with Stable Tracking for Multiple 3D Object Tracking , 2011, IEEE Transactions on Visualization and Computer Graphics.

[6]  Vincent Lepetit,et al.  Keyframe-based modeling and tracking of multiple 3D objects , 2010, 2010 IEEE International Symposium on Mixed and Augmented Reality.

[7]  Anton van den Hengel,et al.  Interactive modelling for AR applications , 2010, 2010 IEEE International Symposium on Mixed and Augmented Reality.

[8]  Gilles Simon.  In-Situ 3D Sketching Using a Video Camera as an Interaction and Tracking Device , 2010, Eurographics.

[9]  Anthony Steed,et al.  Interactive modelling and tracking for mixed and augmented reality , 2006, VRST '06.

[10]  G. Klein,et al.  Parallel Tracking and Mapping for Small AR Workspaces , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[11]  Jianliang Tang,et al.  Complete Solution Classification for the Perspective-Three-Point Problem , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[12]  Andrew W. Fitzgibbon,et al.  Markerless tracking using planar structures in the scene , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[13]  Philip H. S. Torr,et al.  VideoTrace: rapid interactive scene modelling from video , 2007, SIGGRAPH 2007.

[14]  David Nistér,et al.  Scalable Recognition with a Vocabulary Tree , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[15]  Pished Bunnun,et al.  OutlinAR: an assisted interactive model building system with reduced computational effort , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[16]  Hideo Saito,et al.  Virtually augmenting hundreds of real pictures: An approach based on learning, retrieval, and tracking , 2010, 2010 IEEE Virtual Reality Conference (VR).

[17]  Changchang Wu,et al.  SiftGPU : A GPU Implementation of Scale Invariant Feature Transform (SIFT) , 2007 .

[18]  David W. Murray,et al.  Video-rate localization in multiple maps for wearable augmented reality , 2008, 2008 12th IEEE International Symposium on Wearable Computers.

[19]  Vincent Lepetit,et al.  Scalable real-time planar targets tracking for digilog books , 2010, The Visual Computer.