Mixed-Reality World Exploration Using Image-Based Rendering

This article describes a Mixed-Reality (MR) application that superimposes lost buildings of a historical site onto real scenes virtualized using spherical aerial images. The proposed application is set at a UNESCO World Heritage site in Japan, and is based on a novel framework that supports the photorealistic superimposition of virtual objects onto virtualized real scenes. The proposed framework utilizes Image-Based Rendering (IBR), which enables users to freely change their viewpoint in a real-world virtualization constructed using precaptured images. This framework combines the offline rendering of virtual objects and IBR to take advantage of the higher quality of offline rendering without the additional computational cost of online processing; that is, it incurs only the cost of online lightweight IBR, which is simplified through the pregeneration of structured viewpoints (e.g., at grid points).

[1]  Tomás Pajdla,et al.  Multi-view reconstruction preserving weakly-supported surfaces , 2011, CVPR 2011.

[2]  Richard Szeliski,et al.  The geometry-image representation tradeoff for rendering , 2000, Proceedings 2000 International Conference on Image Processing (Cat. No.00CH37101).

[3]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[4]  Philipp Lensing,et al.  Instant indirect illumination for dynamic mixed reality scenes , 2012, 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[5]  Ian D. Bishop,et al.  Integration of augmented reality and GIS: A new approach to realistic landscape visualisation , 2008 .

[6]  Richard Szeliski,et al.  Bundle Adjustment in the Large , 2010, ECCV.

[7]  Gregory J. Ward,et al.  The RADIANCE lighting simulation and rendering system , 1994, SIGGRAPH.

[8]  Takeshi Naemura,et al.  Ray-based creation of photo-realistic virtual world , 1997, Proceedings. International Conference on Virtual Systems and MultiMedia VSMM '97 (Cat. No.97TB100182).

[9]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[10]  Steven M. Seitz,et al.  Multicore bundle adjustment , 2011, CVPR 2011.

[11]  Andrea Bottino,et al.  What's NEXT? An interactive next best view approach , 2006, Pattern Recognit..

[12]  Yizhou Yu,et al.  Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping , 1998, Rendering Techniques.

[13]  OkuraFumio,et al.  Mixed-Reality World Exploration Using Image-Based Rendering , 2015 .

[14]  Naokazu Yokoya,et al.  Aerial full spherical HDR imaging and display , 2014, Virtual Reality.

[15]  Maxime Lhuillier.  Fusion of GPS and structure-from-motion using constrained bundle adjustments , 2011, CVPR 2011.

[16]  André Stork,et al.  Rendering techniques for mixed reality , 2010, Journal of Real-Time Image Processing.

[17]  Robert M. Wolk.  Utilizing Google Earth and Google Sketchup to visualize wind farms , 2008, 2008 IEEE International Symposium on Technology and Society.

[18]  Jan-Michael Frahm,et al.  Detailed Real-Time Urban 3D Reconstruction from Video , 2007, International Journal of Computer Vision.

[19]  Michael Bosse,et al.  Unstructured lumigraph rendering , 2001, SIGGRAPH.

[20]  Y. Koga,et al.  Models of sky radiance distribution and sky luminance distribution , 2004 .

[21]  Takeshi Oishi,et al.  Flying Laser Range Sensor for Large-Scale Site-Modeling and Its Applications in Bayon Digital Archival Project , 2008, International Journal of Computer Vision.

[22]  Greg Humphreys,et al.  Physically Based Rendering, Second Edition: From Theory To Implementation , 2010 .

[23]  Jean-Michel Dischler,et al.  Real-time high-quality View-Dependent Texture Mapping using per-pixel visibility , 2005, GRAPHITE.

[24]  Declan Butler,et al.  Virtual globes: The web-wide world , 2006, Nature.

[25]  Toby Howard,et al.  Per Christensen and Daniel Cohen-Or (Editors) Abstract Rapid Shadow Generation in Real-World Lighting Environments , 2022 .

[26]  Paul E. Debevec,et al.  Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography , 1998, SIGGRAPH '08.

[27]  Steven M. Seitz,et al.  View morphing , 1996, SIGGRAPH.

[28]  Hannes Kaufmann,et al.  High-quality reflections, refractions, and caustics in Augmented Reality and their contribution to visual coherence , 2012, 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[29]  Andrew Chi-Sing Leung,et al.  A compression method for a massive image data set in image-based rendering , 2004, Signal Process. Image Commun..

[30]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[31]  Jitendra Malik,et al.  Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach , 1996, SIGGRAPH.

[32]  Greg Humphreys,et al.  Physically Based Rendering: From Theory to Implementation , 2004 .

[33]  Werner Hartmann,et al.  A real-time shadow approach for an augmented reality application using shadow volumes , 2003, VRST '03.

[34]  Harald Wuest,et al.  Cultural Heritage Layers: Integrating Historic Media in Augmented Reality , 2009, 2009 15th International Conference on Virtual Systems and Multimedia.

[35]  Martin Knecht,et al.  Differential Instant Radiosity for mixed reality , 2010, 2010 IEEE International Symposium on Mixed and Augmented Reality.

[36]  Richard Szeliski,et al.  Building Rome in a day , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[37]  Ronald Azuma,et al.  Recent Advances in Augmented Reality , 2001, IEEE Computer Graphics and Applications.

[38]  Naokazu Yokoya,et al.  Geometric and photometric registration for real-time augmented reality , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[39]  Changchang Wu,et al.  Towards Linear-Time Incremental Structure from Motion , 2013, 2013 International Conference on 3D Vision.

[40]  Naokazu Yokoya,et al.  Extrinsic Camera Parameter Estimation Using Video Images and GPS Considering GPS Positioning Accuracy , 2010, 2010 20th International Conference on Pattern Recognition.

[41]  Richard Pito,et al.  A Solution to the Next Best View Problem for Automated Surface Acquisition , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[42]  Michael Garland,et al.  Surface simplification using quadric error metrics , 1997, SIGGRAPH.

[43]  Simon Gibson,et al.  Interactive Rendering with Real-World Illumination , 2000, Rendering Techniques.

[44]  Zhiqiang Zhang,et al.  Research for 3D visualization of Digital City based on SketchUp and ArcGIS , 2009, Other Conferences.

[45]  George Papagiannakis,et al.  A taxonomy of visualization strategies for cultural heritage applications , 2010, JOCCH.

[46]  Henrik Wann Jensen,et al.  Global Illumination using Photon Maps , 1996, Rendering Techniques.

[47]  Marcus A. Magnor,et al.  Multi-view coding for image-based rendering using 3-D scene geometry , 2003, IEEE Trans. Circuits Syst. Video Technol..

[48]  Naokazu Yokoya,et al.  Augmented telepresence using autopilot airship and omni-directional camera , 2010, 2010 IEEE International Symposium on Mixed and Augmented Reality.

[49]  Naokazu Yokoya,et al.  Fly-through Heijo palace site: augmented telepresence using aerial omnidirectional videos , 2011, SIGGRAPH '11.

[50]  Roberto Scopigno,et al.  ExploreMaps: Efficient construction and ubiquitous exploration of panoramic view graphs of complex 3D environments , 2014, Comput. Graph. Forum.

[51]  Thorsten Grosch.  PanoAR: INTERACTIVE AUGMENTATION OF OMNI-DIRECTIONAL IMAGES WITH CONSISTENT LIGHTING , 2005 .

[52]  Naokazu Yokoya,et al.  Arbitrary Stereoscopic View Generation Using Multiple Omnidirectional Image Sequences , 2010, 2010 20th International Conference on Pattern Recognition.

[53]  Adrien Bousseau,et al.  Rich Intrinsic Image Decomposition of Outdoor Scenes from Multiple Views , 2012, IEEE Transactions on Visualization and Computer Graphics.

[54]  S. B. Kang,et al.  Survey of image-based representations and compression techniques , 2003, IEEE Trans. Circuits Syst. Video Technol..