3D free-viewpoint augmented reality on a second display synchronized with a broadcast program

A commercial 4K/8K satellite broadcasting service will be launched in Japan in December 2018. MPEG media transport (MMT) will be used as the Internet-protocol-based media transport scheme to realize advanced hybrid services over the Internet. In this paper, we describe a model for a 3D free-viewpoint augmented reality (AR) service for second-display devices via the Internet. Using this model, viewers will be able to enjoy a new viewing experience in which a 2D video presentation on a television and 3D free-viewpoint AR presentation on a tablet device are synchronized precisely. We prototype AR content of an imaginary 3D computer-graphics world with some characters and implement a distribution system for the content. In this prototype system, an MMT video streamer transmitted a 2D movie prerendered from a 3D scene. Transmitted video frames over MMT have coordinated universal time (UTC)-based presentation timestamps (PTSs). A motion streamer transmits motion data of the characters in the 3D scene along with the UTC-based PTSs to the Internet. The PTSs are synchronized with the timeline of the 2D movie. Second-display AR receivers render the 3D scene overlaid on top of a table in the real world. On the table, the characters are running and moving in real time according to the received motion data and the PTS.