Optimization-based User Support for Cinematographic Quadrotor Camera Target Framing

To create aesthetically pleasing aerial footage, the correct framing of camera targets is crucial. However, current quadrotor camera tools do not consider the 3D extent of actual camera targets in their optimization schemes and simply interpolate between keyframes when generating a trajectory. This can yield videos with aesthetically unpleasing target framing. In this paper, we propose a target framing algorithm that optimizes the quadrotor camera pose such that targets are positioned at desirable screen locations according to videographic compositional rules and entirely visible throughout a shot. Camera targets are identified using a semi-automatic pipeline which leverages a deep-learning-based visual saliency model. A large-scale perceptual study (N ≈ 500) shows that our method enables users to produce shots with a target framing that is closer to what they intended to create and more or as aesthetically pleasing than with the previous state of the art.

[1]  Tsai-Yen Li,et al.  Real-Time Camera Planning for Navigation in Virtual Environments , 2008, Smart Graphics.

[2]  Sebastian Scherer,et al.  Autonomous aerial cinematography in unstructured environments with learned artistic decision‐making , 2019, J. Field Robotics.

[3]  Dani Lischinski,et al.  Creating and chaining camera moves for quadrotor videography , 2018, ACM Trans. Graph..

[4]  D. Arijon,et al.  Grammar of Film Language , 1976 .

[5]  Ben Shneiderman,et al.  Direct manipulation for comprehensible, predictable and controllable user interfaces , 1997, IUI '97.

[6]  Sebastian Scherer,et al.  Towards a Robust Aerial Cinematography Platform: Localizing and Tracking Moving Targets in Unstructured Environments , 2019, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[7]  Otmar Hilliges,et al.  Airways: Optimization-Based Planning of Quadrotor Trajectories according to High-Level User Goals , 2016, CHI.

[8]  Quentin Galvane,et al.  Automated Cinematography with Unmanned Aerial Vehicles , 2016, WICED@Eurographics.

[9]  Pat Hanrahan,et al.  Towards a Drone Cinematographer: Guiding Quadrotor Cameras using Visual Composition Principles , 2016, ArXiv.

[10]  Marc Christie,et al.  Intuitive and efficient camera control with the toric space , 2015, ACM Trans. Graph..

[11]  Alexander Domahidi,et al.  Real-time planning for automated multi-view drone cinematography , 2017, ACM Trans. Graph..

[12]  Mark O. Riedl,et al.  A Lightweight Intelligent Virtual Cinematography System for Machinima Production , 2007, AIIDE.

[13]  Andrea Cavallaro,et al.  Distributed vision-based flying cameras to film a moving target , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[14]  Chao-Hung Lin,et al.  Efficient camera path planning algorithm for human motion overview , 2011, Comput. Animat. Virtual Worlds.

[15]  Pat Hanrahan,et al.  Generating dynamically feasible trajectories for quadrotor cameras , 2016, ACM Trans. Graph..

[16]  Marc Christie,et al.  The director's lens: an intelligent assistant for virtual cinematography , 2011, ACM Multimedia.

[17]  Bin Wang,et al.  Example-driven virtual cinematography by learning camera behaviors , 2020, ACM Trans. Graph..

[18]  Brian O'Neill,et al.  Creativity support for novice digital filmmaking , 2013, CHI.

[19]  Xin Yang,et al.  ACT: An Autonomous Drone Cinematography System for Action Scenes , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[20]  Patrick Olivier,et al.  Camera Control in Computer Graphics , 2006, Eurographics.

[21]  Otmar Hilliges,et al.  WYFIWYG: Investigating Effective User Support in Aerial Videography , 2018, ArXiv.

[22]  Rita Cunha,et al.  Optimal Trajectory Planning for Autonomous Drone Cinematography , 2019, 2019 European Conference on Mobile Robots (ECMR).

[23]  James C. Lester,et al.  Intelligent multi-shot visualization interfaces for dynamic 3D worlds , 1998, IUI '99.

[24]  Michael Gleicher,et al.  Through-the-lens camera control , 1992, SIGGRAPH.

[25]  Sebastian Scherer,et al.  Can a Robot Become a Movie Director? Learning Artistic Principles for Aerial Cinematography , 2019, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[26]  Leon A. Gatys,et al.  Understanding Low- and High-Level Contributions to Fixation Prediction , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[27]  Marc Christie,et al.  Efficient composition for virtual camera control , 2012, SCA '12.

[28]  Steven M. Drucker,et al.  Intelligent Camera Control in a Virtual Environment , 1994 .

[29]  Pat Hanrahan,et al.  An interactive tool for designing quadrotor camera shots , 2015, ACM Trans. Graph..

[30]  Cristina Stoica,et al.  Quadcopter-performed cinematographic flight plans using minimum jerk trajectories and predictive camera control , 2018, 2018 European Control Conference (ECC).

[31]  James D. Hollan,et al.  Direct Manipulation Interfaces , 1985, Hum. Comput. Interact..

[32]  Marc Christie,et al.  Directing Cinematographic Drones , 2017, ACM Trans. Graph..

[33]  Otmar Hilliges,et al.  Optimizing for aesthetically pleasing quadrotor camera motion , 2018, ACM Trans. Graph..