Shot Orientation Controls for Interactive Cinematography with 360 Video

Virtual reality filmmakers creating 360-degree video currently rely on cinematography techniques that were developed for traditional narrow field of view film. They typically edit together a sequence of shots so that they appear at a fixed-orientation irrespective of the viewer's field of view. But because viewers set their own camera orientation they may miss important story content while looking in the wrong direction. We present new interactive shot orientation techniques that are designed to help viewers see all of the important content in 360-degree video stories. Our viewpoint-oriented technique reorients the shot at each cut so that the most important content lies in the the viewer's current field of view. Our active reorientation technique, lets the viewer press a button to immediately reorient the shot so that important content lies in their field of view. We present a 360-degree video player which implements these techniques and conduct a user study which finds that users spend 5.2-9.5% more time viewing the important points (manually labelled) of the scene with our techniques compared to the traditional fixed-orientation cuts. In practice, 360-degree video creators may label important content, but we also provide an automatic method for determining important content in existing 360-degree videos.

[1]  Don Kimber,et al.  FlyCam: practical panoramic video and automatic camera control , 2000, 2000 IEEE International Conference on Multimedia and Expo. ICME2000. Proceedings. Latest Advances in the Fast Changing World of Multimedia (Cat. No.00TH8532).

[2]  Stephen DiVerdi,et al.  Vremiere: In-Headset Virtual Reality Video Editing , 2017, CHI.

[3]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Kristen Grauman,et al.  Pano2Vid: Automatic Cinematography for Watching 360° Videos , 2017, WICED@Eurographics.

[5]  Ariel Shamir,et al.  Improved seam carving for video retargeting , 2008, SIGGRAPH 2008.

[6]  David Salesin,et al.  Panoramic video textures , 2005, ACM Trans. Graph..

[7]  Jetta Carol Culpepper,et al.  Academic Press Dictionary of Science and Technology , 2000 .

[8]  Abhishek Ranjan,et al.  Automatic camera control using unobtrusive vision and audio tracking , 2010, Graphics Interface.

[9]  Gordon Wetzstein,et al.  Movie editing and cognitive event segmentation in virtual reality video , 2017, ACM Trans. Graph..

[10]  Simon Baker,et al.  Lucas-Kanade 20 Years On: A Unifying Framework , 2004, International Journal of Computer Vision.

[11]  Kristen Grauman,et al.  Pano2Vid: Automatic Cinematography for Watching 360° Videos , 2016, ACCV.

[12]  Kristen Grauman,et al.  Making 360$^{\circ}$ Video Watchable in 2D: Learning Videography for Click Free Viewing , 2017 .

[13]  Yaser Sheikh,et al.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Rainer Lienhart,et al.  Reliable Transition Detection in Videos: A Survey and Practitioner's Guide , 2001, Int. J. Image Graph..

[15]  Tao Wang,et al.  End-to-end text recognition with convolutional neural networks , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[16]  Eugenia M. Kolasinski,et al.  Simulator Sickness in Virtual Environments. , 1995 .

[17]  Peter Robinson,et al.  OpenFace: An open source facial behavior analysis toolkit , 2016, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV).

[18]  Michael Gleicher,et al.  Video retargeting: automating pan and scan , 2006, MM '06.

[19]  J. Cutting,et al.  Attention and the Evolution of Hollywood Film , 2010, Psychological science.

[20]  Mark O. Riedl,et al.  A Lightweight Intelligent Virtual Cinematography System for Machinima Production , 2007, AIIDE.

[21]  Kristen Grauman,et al.  Making 360° Video Watchable in 2D: Learning Videography for Click Free Viewing , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Christopher G. Harris,et al.  A Combined Corner and Edge Detector , 1988, Alvey Vision Conference.

[23]  Bing-Yu Chen,et al.  Outside-In: Visualizing Out-of-Sight Regions-of-Interest in a 360° Video Using Spatial Picture-in-Picture Previews , 2017, UIST.

[24]  Markus H. Gross,et al.  A system for retargeting of streaming video , 2009, ACM Trans. Graph..

[25]  Radomír Mech,et al.  Minimum Barrier Salient Object Detection at 80 FPS , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[26]  David Salesin,et al.  The virtual cinematographer: a paradigm for automatic real-time camera control and directing , 1996, SIGGRAPH.