Augmented reality on cloth with realistic illumination

Augmented reality (AR) is the concept of inserting virtual objects into real scenes. Often, augmentations are aligned with rigid planar objects in the scene. However, a more difficult task is to align non-rigid augmentations with flexible objects like cloth. To address this problem, we present a method to perform real-time flexible augmentations on cloth. Our method involves sparse cloth-tracking in video images using a new vision-based marker system with temporal coherence. We include an image-based method to automatically acquire real world illumination and shadows from the input frame. Non-rigid augmentations are achieved by rendering a textured 2D mesh aligned with the cloth surface, and combining the illumination result. The ability to perform realistic augmentations on cloth leads to applications in fashion, advertising, home decor and entertainment. We demonstrate our cloth augmentations with an application to interactively design T-shirts by demonstrating different virtual logos on a physical shirt in real-time.

[1]  Andrew W. Fitzgibbon,et al.  Markerless tracking using planar structures in the scene , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[2]  Volker Scholz,et al.  Cloth Motion from Optical Flow , 2004, VMV.

[3]  J. Urgen Stauder.  Augmented Reality with Automatic Illumination Control Incorporating Ellipsoidal Models , 1999 .

[4]  D. Bradley,et al.  Augmenting Non-Rigid Objects with Realistic Lighting , 2004 .

[5]  R. K. Shyamasundar,et al.  Introduction to algorithms , 1996 .

[6]  Hideo Saito,et al.  Texture overlay for virtual clothing based on PCA of silhouettes , 2006, 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality.

[7]  A. Fournier,et al.  Common Illumination between Real and Computer Generated Scenes , 1992 .

[8]  Paul Debevec.  Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography , 2008, SIGGRAPH Classes.

[9]  Simon Gibson,et al.  Interactive Rendering with Real-World Illumination , 2000, Rendering Techniques.

[10]  Claus B. Madsen,et al.  The importance of shadows in augmented reality , 2003 .

[11]  Michael Vittrup,et al.  The 6th Annual International Workshop on Presence, Presence 2003, 6-8 October 2003, Aalborg, Denmark , 2003 .

[12]  Guillermo Sapiro,et al.  Image inpainting , 2000, SIGGRAPH.

[13]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[14]  Igor Guskov,et al.  Trackable surfaces , 2003, SCA '03.

[15]  Volker Scholz,et al.  Garment Motion Capture Using Color‐Coded Patterns , 2005, Comput. Graph. Forum.

[16]  George Drettakis,et al.  Interactive Common Illumination for Computer Augmented Reality , 1997, Rendering Techniques.

[17]  Guillermo Sapiro,et al.  Filling-in by joint interpolation of vector fields and gray levels , 2001, IEEE Trans. Image Process..

[18]  Atsuyuki Okabe,et al.  Spatial Tessellations: Concepts and Applications of Voronoi Diagrams , 1992, Wiley Series in Probability and Mathematical Statistics.

[19]  David A. Forsyth,et al.  Retexturing Single Views Using Texture and Shading , 2006, ECCV.

[20]  Yakup Genc,et al.  Marker-less tracking for AR: a learning-based approach , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[21]  Ronald Fedkiw,et al.  Simulation of clothing with folds and wrinkles , 2003, SCA '03.

[22]  Igor Guskov,et al.  Efficient tracking of regular patterns on non-rigid geometry , 2002, Object recognition supported by user interaction for service robots.

[23]  Derek Bradley,et al.  Adaptive Thresholding using the Integral Image , 2007, J. Graph. Tools.

[24]  W. Heidrich,et al.  Texture Replacement of Garments in Monocular Video Sequences , 2022 .

[25]  Vincent Lepetit,et al.  Augmenting deformable objects in real-time , 2005, Fourth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'05).

[26]  Werner Hartmann,et al.  A real-time shadow approach for an augmented reality application using shadow volumes , 2003, VRST '03.

[27]  Vincent Lepetit,et al.  Real-time nonrigid surface detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[28]  Ivan Poupyrev,et al.  The MagicBook - Moving Seamlessly between Reality and Virtuality , 2001, IEEE Computer Graphics and Applications.

[29]  Wolfgang Heidrich,et al.  Cloth Motion Capture , 2003, Comput. Graph. Forum.

[30]  Toby Howard,et al.  Rapid Shadow Generation in Real-World Lighting Environments , 2003, Rendering Techniques.

[31]  Éric Marchand,et al.  A real-time tracker for markerless augmented reality , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..