AR-ia: Volumetric Opera for Mobile Augmented Reality

Motivated by recent availability of augmented and virtual reality platforms, we tackle the challenging problem of immersive storytelling experiences on mobile devices. In particular, we show an end-to-end system to generate 3D assets that enable real-time rendering of an opera on high end mobile phones. We call our system AR-ia and in this paper we walk through the main components and technical challenges of such a system, showing how to deliver an immersive mixed reality experience in every user’s living room.

[1]  Alvaro Collet,et al.  High-quality streamable free-viewpoint video , 2015, ACM Trans. Graph..

[2]  Kun Zhou,et al.  Iso-charts: stretch-driven mesh parameterization using spectral analysis , 2004, SGP '04.

[3]  Charles T. Loop,et al.  Holoportation: Virtual 3D Teleportation in Real-time , 2016, UIST.

[4]  Pushmeet Kohli,et al.  Fusion4D , 2016, ACM Trans. Graph..

[5]  Shahram Izadi,et al.  Motion2fusion , 2017, ACM Trans. Graph..

[6]  Shahram Izadi,et al.  UltraStereo: Efficient Learning-Based Matching for Active Stereo Systems , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Shahram Izadi,et al.  Low Compute and Fully Parallel Computer Vision with HashMatch , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[8]  Paul E. Debevec,et al.  Acquiring the reflectance field of a human face , 2000, SIGGRAPH.

[9]  David Kim,et al.  The need 4 speed in real-time dense visual tracking , 2018, ACM Trans. Graph..