Interactive 3D content insertion in images for multimedia applications

This article addresses the problem of creating interactive mixed reality applications where virtual objects interact in images of real world scenarios. This is relevant to create games and architectural or space planning applications that interact with visual elements in the images such as walls, floors and empty spaces. These scenarios are intended to be captured by the users with regular cameras or using previously taken photographs. Introducing virtual objects in photographs presents several challenges, such as pose estimation and the creation of a visually correct interaction between virtual objects and the boundaries of the scene. The two main research questions addressed in this article include, the study of the feasibility of creating interactive augmented reality (AR) applications where virtual objects interact in a real world scenario using the image detected high-level features and, also, verifying if untrained users are capable and motivated enough to perform AR initialization steps. The proposed system detects the scene automatically from an image with additional features obtained using basic annotations from the user. This operation is significantly simple to accommodate the needs of non-expert users. The system analyzes one or more photos captured by the user and detects high-level features such as vanishing points, floor and scene orientation. Using these features it will be possible to create mixed and augmented reality applications where the user interactively introduces virtual objects that blend with the picture in real time and respond to the physical environment. To validate the solution several system tests are described and compared using available external image datasets.

[1]  Rafael Grompone von Gioi,et al.  On Straight Line Segment Detection , 2008, Journal of Mathematical Imaging and Vision.

[2]  Gilles Simon.  Automatic online walls detection for immediate use in AR tasks , 2006, 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality.

[3]  David A. Forsyth,et al.  Rendering synthetic objects into legacy photographs , 2011, ACM Trans. Graph..

[4]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[5]  David A. Forsyth,et al.  Recovering free space of indoor scenes from a single image , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[6]  Pierre Fite-Georgel,et al.  Is there a reality in Industrial Augmented Reality? , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[7]  Dieter Schmalstieg,et al.  Multiple target detection and tracking with guaranteed framerates on mobile phones , 2009, 2009 8th IEEE International Symposium on Mixed and Augmented Reality.

[8]  Richard Szeliski,et al.  Manhattan-world stereo , 2009, CVPR.

[9]  Trung-Nghia Le,et al.  Augmented media for traditional magazines , 2012, SoICT.

[10]  Dieter Schmalstieg,et al.  Real-Time Detection and Tracking for Augmented Reality on Mobile Phones , 2010, IEEE Transactions on Visualization and Computer Graphics.

[11]  Nuno Correia,et al.  Photo-based Multimedia Applications using Image Features Detection , 2013, GRAPP/IVAPP.

[12]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[13]  Alexei A. Efros,et al.  Recovering Surface Layout from an Image , 2007, International Journal of Computer Vision.

[14]  Andrew W. Fitzgibbon,et al.  Markerless tracking using planar structures in the scene , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[15]  Stephen Gould,et al.  Single image depth estimation from predicted semantic labels , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[16]  Joseph Schlecht,et al.  Sampling bedrooms , 2011, CVPR 2011.

[17]  Nuno Correia,et al.  Magnetic augmented reality: virtual objects in your space , 2012, AVI.

[18]  Carsten Rother.  A new approach to vanishing point detection in architectural environments , 2002, Image Vis. Comput..

[19]  Fumio Kishino,et al.  Augmented reality: a class of displays on the reality-virtuality continuum , 1995, Other Conferences.

[20]  Dima Damen,et al.  Integrating 3D object detection, modelling and tracking on a mobile phone , 2012, 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[21]  Veronica Teichrieb,et al.  Texture-less planar object detection and pose estimation using Depth-Assisted Rectification of Contours , 2012, 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[22]  Takeo Kanade,et al.  Estimating Spatial Layout of Rooms using Volumetric Reasoning about Objects and Surfaces , 2010, NIPS.

[23]  Ian D. Reid,et al.  Single View Metrology , 2000, International Journal of Computer Vision.

[24]  Andrew Blake,et al.  "GrabCut" , 2004, ACM Trans. Graph..

[25]  Éric Marchand,et al.  Toward augmenting everything: Detecting and tracking geometrical features on planar objects , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[26]  Derek Hoiem,et al.  Recovering the spatial layout of cluttered rooms , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[27]  Richard Szeliski,et al.  Manhattan-world stereo , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[28]  Marie-Odile Berger,et al.  Pose Estimation for Planar Structures , 2002, IEEE Computer Graphics and Applications.

[29]  Rafael Grompone von Gioi,et al.  Multisegment Detection , 2007, 2007 IEEE International Conference on Image Processing.

[30]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  David A. Forsyth.  Understanding pictures of rooms: technical perspective , 2013, CACM.

[32]  Hongbin Zha,et al.  Vanishing point detection using cascaded 1D Hough Transform from single images , 2012, Pattern Recognit. Lett..

[33]  Anne Bationo Tillon,et al.  Mobile augmented reality in the museum: Can a lace-like technology take you closer to works of art? , 2011, 2011 IEEE International Symposium on Mixed and Augmented Reality - Arts, Media, and Humanities.

[34]  Alan L. Yuille,et al.  Manhattan World: compass direction from a single image by Bayesian inference , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[35]  Nuno Correia,et al.  Dynamic Insertion of Virtual Objects in Photographs , 2013, Int. J. Creative Interfaces Comput. Graph..

[36]  Dieter Schmalstieg,et al.  Indoor navigation with mixed reality world-in-miniature views and sparse localization on mobile devices , 2012, AVI.

[37]  H. Seidel,et al.  Pattern-aware Deformation Using Sliding Dockers , 2011, SIGGRAPH 2011.

[38]  G. Klein,et al.  Parallel Tracking and Mapping for Small AR Workspaces , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[39]  Alexei A. Efros,et al.  From 3D scene geometry to human workspace , 2011, CVPR 2011.

[40]  Ronald Azuma,et al.  A survey of augmented reality" Presence: Teleoperators and virtual environments , 1997 .

[41]  Stephen Gould,et al.  Decomposing a scene into geometric and semantically consistent regions , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[42]  Christopher M. Brown,et al.  Interactive Augmented Reality , 1998 .

[43]  Yuri Boykov,et al.  Globally optimal segmentation of multi-region objects , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[44]  Martial Hebert,et al.  3-D scene analysis via sequenced predictions over points and regions , 2011, 2011 IEEE International Conference on Robotics and Automation.

[45]  Changsheng Xu,et al.  A generic virtual content insertion system based on visual attention analysis , 2008, ACM Multimedia.

[46]  T. Kanade,et al.  Geometric reasoning for single image structure recovery , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.