Compelling virtual reality experiences require high quality imagery as well as head motion with six degrees of freedom. Most existing systems limit the motion of the viewer (prerecorded fixed position 360 video panoramas), or are limited in realism, e.g. video game quality graphics rendered in real-time on low powered devices. We propose a solution for presenting movie quality graphics to the user while still allowing the sense of presence afforded by free viewpoint head motion. By transforming offline rendered movie content into a novel immersive deep media representation, we display the content in real-time according to the tracked head pose. For each frame, we generate a set of 360-degree images (colors and depths) using cameras placed in selected locations within a small view volume surrounding a central viewing position. We employ a parallax masking technique which minimizes the rendering work required for the additionally visible surfaces in viewing locations around the main viewpoint. At run-time, a decompression and rendering algorithm fetches the appropriate surface data in real-time and projects them to the eye positions as the user moves within the tracked view volume. To further illustrate this ability for interactivity and embodiment within VR movies, we track the full upper body using our sparse sensor motion capture solver allowing users to see themselves in the virtual world. Here, both head and upper body are tracked in realtime using data from IMU (Inertial Measurement Unit) and EMG (Electromyogram) sensors. Our real-time solver, Triduna Live uses a physics-based approach to robustly estimate pose from a few sensors. Hand gesture and object grasping motions are detected from the EMG data and combined with the tracked body position to control gameplay seamlessly integrated within the deep media environment.
[1]
Paul E. Debevec,et al.
Spherical light field environment capture for virtual reality using a motorized pan/tilt head and offset camera
,
2015,
SIGGRAPH Posters.
[2]
Manuel Menezes de Oliveira Neto,et al.
Relief mapping of non-height-field surface details
,
2006,
I3D '06.
[3]
Da-Yuan Huang,et al.
Moving around in virtual space with spider silk
,
2015,
SIGGRAPH Emerging Technologies.
[4]
Markus H. Gross,et al.
Online view sampling for estimating depth from light fields
,
2015,
2015 IEEE International Conference on Image Processing (ICIP).
[5]
Max Rheiner.
Birdly an attempt to fly
,
2014,
SIGGRAPH '14.