Dynamic Insertion of Virtual Objects in Photographs

Introducing virtual objects in photographs or video sequences presents several challenges, such as the pose estimation and the visually correct interaction boundaries of such objects. In this article a framework for the introduction of virtual objects in user-captured photos is discussed. Furthermore, the introduced virtual objects should be interactive and respond to real physical environments. The proposed detection system is semi-automatic and thus depends on the user to obtain the elements it needs. This operation should be significantly simple to accommodate the needs of a non-expert user. The system analyses a photo taken by the user and detects high-level features such as vanishing points, floor and scene orientation. Using these features it will be possible to create virtual mixed and augmented reality applications where the user takes one or more photos of a certain place and interactively introduces virtual objects or elements that blend with the picture in real time. This article discusses the techniques required to acquire images and information about the scenario involving the user. To demonstrate the framework, a proof-of-concept implementation is presented. This implementation was used to conduct a user study regarding the evaluation of the reliability of the concept. The presented results show a high reliability in the scene detection and that users are able and motivated to use this type of systems.

[1]  Joseph Schlecht,et al.  Sampling bedrooms , 2011, CVPR 2011.

[2]  G. Klein,et al.  Parallel Tracking and Mapping for Small AR Workspaces , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[3]  Tomoya Ishikawa,et al.  Interactive 3-D indoor modeler for virtualizing service fields , 2011, Virtual Reality.

[4]  Jitendra Malik,et al.  Inferring spatial layout from a single image via depth-ordered grouping , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[5]  Michael G. Strintzis,et al.  3D Modeling and Animation: Synthesis and Analysis Techniques for the Human Body , 2004 .

[6]  Carsten Rother.  A new approach to vanishing point detection in architectural environments , 2002, Image Vis. Comput..

[7]  Chieh-Li Chen,et al.  Tennis real play: an interactive tennis game with models from real videos , 2011, MM '11.

[8]  Gilles Simon.  Automatic online walls detection for immediate use in AR tasks , 2006, 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality.

[9]  David A. Forsyth,et al.  Rendering synthetic objects into legacy photographs , 2011, ACM Trans. Graph..

[10]  Tsuhan Chen,et al.  Active learning for piecewise planar 3D reconstruction , 2011, CVPR 2011.

[11]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[12]  Ian D. Reid,et al.  Single View Metrology , 2000, International Journal of Computer Vision.

[13]  Alexei A. Efros,et al.  From 3D scene geometry to human workspace , 2011, CVPR 2011.

[14]  Takeo Kanade,et al.  Geometric reasoning for single image structure recovery , 2009, CVPR.

[15]  Dieter Schmalstieg,et al.  Real-Time Detection and Tracking for Augmented Reality on Mobile Phones , 2010, IEEE Transactions on Visualization and Computer Graphics.

[16]  Alan L. Yuille,et al.  Manhattan World: compass direction from a single image by Bayesian inference , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[17]  Peter Simon,et al.  Augmenting experiences — A bridge between two universities , 2011, 2011 IEEE International Symposium on Mixed and Augmented Reality - Arts, Media, and Humanities.

[18]  Anne Bationo Tillon,et al.  Mobile augmented reality in the museum: Can a lace-like technology take you closer to works of art? , 2011, 2011 IEEE International Symposium on Mixed and Augmented Reality - Arts, Media, and Humanities.

[19]  Elif E. Ayiter.  Becoming Creative through Self Observation: A (Second Order) Cybernetic Learning Strategy for the Metaverse , 2011, Int. J. Art Cult. Des. Technol..

[20]  Nuno Correia,et al.  Photo-based Multimedia Applications using Image Features Detection , 2013, GRAPP/IVAPP.

[21]  Kin Choong Yow,et al.  Robust matching of building facades under large viewpoint changes , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[22]  Dew Harrison,et al.  Digital Media and Technologies for Virtual Artistic Spaces , 2013 .

[23]  Andrew W. Fitzgibbon,et al.  Markerless tracking using planar structures in the scene , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[24]  Derek Hoiem,et al.  Recovering the spatial layout of cluttered rooms , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[25]  Henry Been-Lirn Duh,et al.  Handheld AR games — A triarchic conceptual design framework , 2011, 2011 IEEE International Symposium on Mixed and Augmented Reality - Arts, Media, and Humanities.

[26]  Takeo Kanade,et al.  Estimating Spatial Layout of Rooms using Volumetric Reasoning about Objects and Surfaces , 2010, NIPS.

[27]  Jan-Michael Frahm,et al.  Detailed Real-Time Urban 3D Reconstruction from Video , 2007, International Journal of Computer Vision.

[28]  Ken Perlin,et al.  ClayVision: the (elastic) image of the city , 2012, CHI.

[29]  Angel D. Sappa,et al.  Advances in Vision-Based Human Body Modeling , 2004 .

[30]  Sherry Mayo,et al.  A Model for a Collective Aesthetic Consciousness , 2011, Int. J. Art Cult. Des. Technol..

[31]  Richard Szeliski,et al.  Piecewise planar stereo for image-based rendering , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[32]  Andrew W. Fitzgibbon,et al.  KinectFusion: real-time dynamic 3D surface reconstruction and interaction , 2011, SIGGRAPH '11.

[33]  Luc Van Gool,et al.  SURF: Speeded Up Robust Features , 2006, ECCV.

[34]  Ashutosh Saxena,et al.  Make3D: Learning 3D Scene Structure from a Single Still Image , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[35]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[36]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[37]  Nuno Correia,et al.  Magnetic augmented reality: virtual objects in your space , 2012, AVI.