Pmomo: Projection Mapping on Movable 3D Object

We introduce Pmomo (acronym of projection mapping on movable object), a dynamic projection mapping system that tracks the 6-DOF position of real-world object, and shades it with virtual 3D contents by projection. The system can precisely lock the projection on the moving object in real-time, even the one with complex geometry. Based on depth camera, we developed a novel and robust tracking method that samples the structure of the object into low-density point cloud, then performs an adaptive searching scheme for the registration procedure. As a fully interactive system, our method can handle both internal and external complex occlusions, and can quickly track back the object even when losing track. In order to further improve the realism of the projected virtual textures, our system innovatively culls occlusions away from projection, which is achieved by a facet-covering method. As a result, the Pmomo system enables the possibility of new interactive Augmented Reality applications that require high-quality dynamic projection effect.

[1]  Paul J. Besl,et al.  A Method for Registration of 3-D Shapes , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[3]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[4]  Marc Levoy,et al.  Efficient variants of the ICP algorithm , 2001, Proceedings Third International Conference on 3-D Digital Imaging and Modeling.

[5]  Ruigang Yang,et al.  Automatic and Continuous Projector Display Surface Estimation Using Everyday Imagery , 2001, WSCG.

[6]  Ramesh Raskar,et al.  Dynamic shader lamps : painting on movable objects , 2001, Proceedings IEEE and ACM International Symposium on Augmented Reality.

[7]  Markus H. Gross,et al.  Embedding imperceptible patterns into projected images for simultaneous acquisition and display , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[8]  Rahul Sukthankar,et al.  Tracking Locations of Moving Hand-Held Displays Using Projected Light , 2005, Pervasive.

[9]  Nikolaus F. Troje,et al.  Paper windows: interaction techniques for digital paper , 2005, CHI.

[10]  Henry Fuchs,et al.  Real-Time Projector Tracking on Complex Geometry Using Ordinary Imagery , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Robert Bridson,et al.  Fast Poisson disk sampling in arbitrary dimensions , 2007, SIGGRAPH '07.

[12]  Scott E. Hudson,et al.  Foldable interactive displays , 2008, UIST '08.

[13]  Raimund Dachselt,et al.  PaperLens: advanced magic lens interaction above the tabletop , 2009, ITS '09.

[14]  Direct Estimation of Nonrigid Registrations with Image-Based Self-Occlusion Reasoning , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Chris Harrison,et al.  OmniTouch: wearable multitouch interaction everywhere , 2011, UIST.

[16]  Max Mühlhäuser,et al.  Xpaaand: interaction techniques for rollable displays , 2011, CHI.

[17]  Daniel G. Aliaga,et al.  Perceptually Based Appearance Modification for Compliant Appearance Editing , 2011, Comput. Graph. Forum.

[18]  Andrew W. Fitzgibbon,et al.  KinectFusion: Real-time dense surface mapping and tracking , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[19]  Andrew W. Fitzgibbon,et al.  KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera , 2011, UIST.

[20]  Otmar Hilliges,et al.  Steerable augmented reality with the beamatron , 2012, UIST.

[21]  Daniel G. Aliaga,et al.  Fast high-resolution appearance editing using superimposed projections , 2012, TOGS.

[22]  Jürgen Steimle,et al.  FoldMe: interacting with double-sided foldable displays , 2012, Tangible and Embedded Interaction.

[23]  Maneesh Agrawala,et al.  3D puppetry: a kinect-based interface for 3D animation , 2012, UIST.

[24]  Andrew Wilson,et al.  MirageTable: freehand interaction on a projected augmented reality tabletop , 2012, CHI.

[25]  Reinhard Koch,et al.  Direct Model-Based Tracking of 3D Object Deformations in Depth and Color Video , 2012, International Journal of Computer Vision.

[26]  Masatoshi Ishikawa,et al.  Lumipen: Projection-Based Mixed Reality for Dynamic Objects , 2012, 2012 IEEE International Conference on Multimedia and Expo.

[27]  Gabriel Taubin,et al.  Simple, Accurate, and Robust Projector-Camera Calibration , 2012, 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission.

[28]  Greg Welch,et al.  A general approach for closed-loop registration in AR , 2012, 2013 IEEE Virtual Reality (VR).

[29]  Pattie Maes,et al.  Flexpad: highly flexible bending interactions for projected handheld displays , 2013, CHI.

[30]  Leonidas J. Guibas,et al.  Object tracking in the presence of occlusions using multiple cameras: A sensor network approach , 2013, TOSN.

[31]  Hans-Werner Gellersen,et al.  Cooperative augmentation of mobile smart objects with projected displays , 2013, TIIS.

[32]  Ichiro Sakuma,et al.  Augmented Reality Navigation With Automatic Marker-Free Image Registration Using 3-D Image Overlay for Dental Surgery , 2014, IEEE Transactions on Biomedical Engineering.

[33]  Hrvoje Benko,et al.  Dyadic projected spatial augmented reality , 2014, UIST.

[34]  Christoph Resch,et al.  Sticky projections — A new approach to interactive shader lamp tracking , 2014, 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[35]  Aditi Majumder,et al.  High-resolution lighting of 3D reliefs using a network of projectors and cameras , 2015, 2015 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON).

[36]  Justus Thies,et al.  Real-time pixel luminance optimization for dynamic multi-projection mapping , 2015, ACM Trans. Graph..

[37]  Dieter Fox,et al.  DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Koichi Hashimoto,et al.  Sticky projection mapping: 450-fps tracking projection onto a moving planar surface , 2015, SIGGRAPH Asia Emerging Technologies.