View enhancement in stereo streams assisted by object model

Object registration between virtual model and real object in image sequences is a challenging task. With partial reconstruction of the real object from binocular images, we approach the problem in a 3D-3D way. The system we developed includes three major components: reconstruction, 3D-3D object registration and visual augmentation from object models. For reconstruction, a perspective stereo-motion based approach is used to recover the projective depths of features, a surface smoothing algorithm of moving least squares under perspective assumption enhances the accuracy of the reconstructions. For 3D-3D object registration, we use model template search to obtain an initial guess of the pose transformation from the model to image data, then Iterative Closest Point algorithm is applied to refine the initial guess. Experiments on real sequences show the effectiveness of the proposed approach.

[1]  Vincent Lepetit,et al.  Fully automated and stable registration for augmented reality applications , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[2]  Andrew J. Davison,et al.  DTAM: Dense tracking and mapping in real-time , 2011, 2011 International Conference on Computer Vision.

[3]  Andrew J. Davison,et al.  Live dense reconstruction with a single moving camera , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[4]  P. Lancaster,et al.  Surfaces generated by moving least squares methods , 1981 .

[5]  David G. Lowe,et al.  What and Where: 3D Object Recognition with Accurate Pose , 2006, Toward Category-Level Object Recognition.

[6]  Jan-Michael Frahm,et al.  From structure-from-motion point clouds to fast location recognition , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Thabo Beeler,et al.  High-quality single-shot capture of facial geometry , 2010, ACM Trans. Graph..

[8]  Marc Levoy,et al.  Efficient variants of the ICP algorithm , 2001, Proceedings Third International Conference on 3-D Digital Imaging and Modeling.

[9]  Nico Blodow,et al.  Fast Point Feature Histograms (FPFH) for 3D registration , 2009, 2009 IEEE International Conference on Robotics and Automation.

[10]  Horst Bischof,et al.  From structure-from-motion point clouds to fast location recognition , 2009, CVPR.

[11]  Matthew Turk,et al.  Location-based augmented reality on mobile phones , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[12]  Ronald Chung,et al.  Dense reconstruction by stereo-motion under perspective camera model , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[13]  Torsten Sattler,et al.  Fast image-based localization using direct 2D-to-3D matching , 2011, 2011 International Conference on Computer Vision.

[14]  Li Xu,et al.  Consistent Binocular Depth and Scene Flow with Chained Temporal Profiles , 2013, International Journal of Computer Vision.

[15]  Daniel Cremers,et al.  LSD-SLAM: Large-Scale Direct Monocular SLAM , 2014, ECCV.

[16]  Friedrich Fraundorfer,et al.  Visual Odometry Part I: The First 30 Years and Fundamentals , 2022 .

[17]  Shahram Izadi,et al.  MonoFusion: Real-time 3D reconstruction of small scenes with a single web camera , 2013, 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[18]  Steven Zhiying Zhou,et al.  Positioning, tracking and mapping for outdoor augmentation , 2010, 2010 IEEE International Symposium on Mixed and Augmented Reality.

[19]  Hugh F. Durrant-Whyte,et al.  Simultaneous localization and mapping: part I , 2006, IEEE Robotics & Automation Magazine.

[20]  Pittsburgh,et al.  The MOPED framework: Object recognition and pose estimation for manipulation , 2011 .

[21]  Thabo Beeler,et al.  High-quality single-shot capture of facial geometry , 2010, SIGGRAPH 2010.

[22]  Jorge Vicente Lopes da Silva,et al.  3D Scanning Using RGBD Imaging Devices: A Survey , 2015 .