MR360 interactive: playing with digital creatures in 360° videos

We present "MR360 Interactive", interactive mixed reality (MR) experiences using pre-recorded and live streaming 360° videos (360-video) shown in head mounted displays. We developed the MR360 toolkit, an interface to create interactive MR content using 360-video. The toolkit detects salient lights in the 360-video and casts realistic shadows. Image based lighting is perceptually optimized to provide fast results. Real-time differential rendering obtains a composition between the virtual objects and the real-world background. We present two applications of our toolkit: a VR experience using pre-recorded 360-video, and the MR Stage, an MR experience using live streaming 360-video. In both applications, participants are interacting with digital creatures with high presence in 360-video.