Augmenting Non-Rigid Objects with Realistic Lighting

In augmented reality, virtual objects are added to the real world by superimposing them onto a video stream or a head-mounted display in real-time. Typical augmented reality applications track 2D patterns on rigid planar objects in order to acquire the pose of the camera in the scene. Rigid augmentations are then performed on the planar objects. We present a method to track non-rigid objects such as cloth, and to perform flexible augmentations on the cloth while stretching and rippling it in real-time, using a single camera. In addition, we present a simple technique to apply real-world illumination to the augmentation, which greatly improves perception and realism. Results show believable real-time augmentations of a non-rigid object, even under extreme lighting conditions.

[1]  David G. Lowe,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004, International Journal of Computer Vision.

[2]  Anthony J. Yezzi,et al.  Anti-geometric diffusion for adaptive thresholding and fast segmentation , 2003, IEEE Trans. Image Process..

[3]  Éric Marchand,et al.  A real-time tracker for markerless augmented reality , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[4]  Werner Hartmann,et al.  A real-time shadow approach for an augmented reality application using shadow volumes , 2003, VRST '03.

[5]  Wolfgang Heidrich,et al.  Cloth Motion Capture , 2003, SIGGRAPH '03.

[6]  Toby Howard,et al.  Per Christensen and Daniel Cohen-Or (Editors) Abstract Rapid Shadow Generation in Real-World Lighting Environments , 2022 .

[7]  Olga Veksler,et al.  Fast variable window for stereo correspondence using integral images , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[8]  Igor Guskov,et al.  Efficient tracking of regular patterns on non-rigid geometry , 2002, Object recognition supported by user interaction for service robots.

[9]  I. Guskov,et al.  Real-time tracking of quad-marked surfaces , 2002, Workshop on Motion and Video Computing, 2002. Proceedings..

[10]  Yakup Genc,et al.  Marker-less tracking for AR: a learning-based approach , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[11]  Fan Xiao,et al.  What is the best fiducial? , 2002, The First IEEE International Workshop Agumented Reality Toolkit,.

[12]  Takeshi Naemura,et al.  Virtual Shadows in Mixed Reality Environment Using Flashlight-like Devices , 2002 .

[13]  Ronald Azuma,et al.  Recent Advances in Augmented Reality , 2001, IEEE Computer Graphics and Applications.

[14]  Wolfgang Broll,et al.  Illuminating the mixed reality stage: applying complex lighting conditions to AR , 2001, Proceedings IEEE and ACM International Symposium on Augmented Reality.

[15]  Michael J. Jones,et al.  Robust Real-Time Face Detection , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[16]  Xiao-Ping Zhang,et al.  Segmentation of bright targets using wavelets and adaptive thresholding , 2001, IEEE Trans. Image Process..

[17]  Andrew W. Fitzgibbon,et al.  Markerless tracking using planar structures in the scene , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[18]  George Drettakis,et al.  Interactive Virtual Relighting of Real Scenes , 2000, IEEE Trans. Vis. Comput. Graph..

[19]  Simon Gibson,et al.  Interactive Rendering with Real-World Illumination , 2000, Rendering Techniques.

[20]  Pierre Poulin,et al.  Interactive Virtual Relighting and Remodeling of Real Scenes , 1999, Rendering Techniques.

[21]  Paul E. Debevec,et al.  Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography , 1998, SIGGRAPH '08.

[22]  Hui Zhu,et al.  Adaptive thresholding by variational method , 1998, IEEE Trans. Image Process..

[23]  George Drettakis,et al.  Interactive Common Illumination for Computer Augmented Reality , 1997, Rendering Techniques.

[24]  A. Fournier,et al.  Common Illumination between Real and Computer Generated Scenes , 1992 .

[25]  Franklin C. Crow,et al.  Summed-area tables for texture mapping , 1984, SIGGRAPH.

[26]  HENRI GOURAUD,et al.  Continuous Shading of Curved Surfaces , 1971, IEEE Transactions on Computers.

[27]  Claus B. Madsen,et al.  The importance of shadows in augmented reality , 2003 .

[28]  Leonid Zhukov,et al.  Direct Pattern Tracking On Flexible Geometry , 2002, WSCG.

[29]  Bernhard P. Wrobel.  Multiple View Geometry in Computer Vision , 2001, Künstliche Intell..

[30]  J. Urgen Stauder.  Augmented Reality with Automatic Illumination Control Incorporating Ellipsoidal Models , 1999 .

[31]  Pierre Wellner,et al.  Adaptive Thresholding for the DigitalDesk , 1993 .

[32]  I. Guskov,et al.  Eurographics/siggraph Symposium on Computer Animation (2003) Trackable Surfaces , 2022 .