Mixed Reality Storytelling Environments Based on Tangible User Interface: Take Origami as an Example

This paper presents a mixed reality storytelling system, which takes handicrafts as tangible interaction tools. Via the system, users can learn handicraft and then use the handicraft pieces to design, create and tell stories with HoloLens iteratively. In order to overcome the limitations of HoloLens gesture interaction, the system uses hand tracking with Kinect to implement a touch-like effect on the desktop. User study shows that our system has good usability, and it is welcomed by users. In addition, it can stimulate users interest in handicraft and storytelling, and even promote parent-child interaction effectively.