3D live humans in mixed reality entertainment

We present a system that introduces live captured 3-d content into augmented reality. For each frame we estimate the transformation between the users point of view and a card marker. We then generate a novel or “virtual” view of the live subject from the same perspective. This “virtual view” is generated based on measurements from fifteen cameras that surround the subject. A fast shape-from-silhouette algorithm reconstructs a 3-d model of the live subject from the user’s perspective and re-samples the original images to generate the novel view. This is then rendered into the user’s world at interactive speeds. The result gives the strong impression that the subject is a real part of the 3-D scene. We present a novel mixed reality entertainment application in which users apply tangible computing techniques to interact with a collaborator in a virtual space.