AR Record&Replay: situated compositing of video content in mobile augmented reality

In this paper we present a novel approach to record and replay video content composited in-situ with a live view of the real environment. Our real-time technique works on mobile phones, and uses a panorama-based tracker to create visually seamless and spatially registered overlay of video content. We apply a temporal foreground-background segmentation of video footage and show how the segmented information can be precisely registered in real-time in the camera view of a mobile phone. We describe the user interface and the video post effects implemented in our prototype as well as our approach with a skateboard training application. Our technique can also be used with online video material and supports the creation of augmented situated documentaries.

[1]  Blair MacIntyre,et al.  Integrating 2-D Video Actors into 3-D Augmented-Reality Systems , 2002, Presence: Teleoperators & Virtual Environments.

[2]  Jens Grubert,et al.  Augmented Reality Browser Survey , 2012 .

[3]  Andrew Blake,et al.  "GrabCut" , 2004, ACM Trans. Graph..

[4]  Lorenz Rogge,et al.  Space-time visual effects as a post-production process , 2010, 3DVP '10.

[5]  Steven M. Seitz,et al.  Photo tourism: exploring photo collections in 3D , 2006, ACM Trans. Graph..

[6]  Hirokazu Kato,et al.  3D live: real time captured content for mixed reality , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[7]  Marc Pollefeys,et al.  Unstructured video-based rendering: interactive exploration of casually captured videos , 2010, SIGGRAPH 2010.

[8]  Blair MacIntyre,et al.  Ghosts in the Machine: Integrating 2D Video Actors into a 3D AR System , 2001 .

[9]  Steven K. Feiner,et al.  Visualizing and navigating complex situated hypermedia in augmented and virtual reality , 2006, 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality.

[10]  Steven K. Feiner,et al.  Situated documentaries: embedding multimedia presentations in the real world , 1999, Digest of Papers. Third International Symposium on Wearable Computers.

[11]  Pished Bunnun,et al.  OutlinAR: an assisted interactive model building system with reduced computational effort , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[12]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[13]  Dieter Schmalstieg,et al.  Real-time panoramic mapping and tracking on mobile phones , 2010, 2010 IEEE Virtual Reality Conference (VR).

[14]  Dieter Schmalstieg,et al.  Mobile Phones as a Platform for Augmented Reality , 2008 .

[15]  Blair MacIntyre,et al.  Three Angry Men: An Augmented-Reality Experiment in Point-of-View Drama , 2003 .

[16]  Suya You,et al.  Real-Time Object Tracking for Augmented Reality Combining Graph Cuts and Optical Flow , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[17]  Dieter Schmalstieg,et al.  Pose tracking from natural features on mobile phones , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[18]  Oliver Schreer,et al.  Proceedings of the 1st international workshop on 3D video processing , 2010, MM 2010.