Horizontal-to-Vertical Video Conversion

At this blooming age of social media and mobile platform, mass consumers are migrating from horizontal video to vertical contents delivered on hand-held devices. Accordingly, revitalizing the exposure of horizontal video becomes vital and urgent, which is hereby settled, for the first time, with our automated horizontal-to-vertical (abbreviated as H2V) video conversion framework. Essentially, the H2V framework performs subjectpreserving video cropping instantiated in the proposed RankSS module. Rank-SS incorporates object detection to discover candidate subjects, from which we select the primary subjectto-preserve leveraging location, appearance, and salient cues in a convolutional neural network. Besides converting horizontal videos to vertically by cropping around the selected subject, automatic shot detection and multi-object tracking are also integrated in the H2V framework to accommodate long and complex videos. In addition, for the development of H2V systems, we publicize an H2V-142K dataset containing 125 videos (132K frames) and 9,500 cover images annotated with primary subject bounding boxes. On H2V-142K and public object detection datasets, our method demonstrates superior subject selection accuracy comparing to related solutions. Beyond that, our H2V framework is also industrially-deployed hosting millions of daily active users and exhibits favorable H2V conversion performance. Upon publicizing this dataset as well as our approach, we wish to pave the way for more horizontal-to-vertical video conversion solutions to come.