A Simple Learning Framework for Large Vocabulary Video Object Detection

Applying deep learning in the video domain fundamentally suffers from the data-hungry issue, and the situation will become even more severe for more complex and challenging tasks. One promising direction we believe is leveraging already well-curated large-scale image data to complement the insufficient video data. However, jointly using multiple datasets [14], image and video labels, leads to several issues, detailed below. In this paper, we investigate the new problem of large vocabulary tracking, one of the essential milestones for dynamic world understanding AI agents. The task naturally lacks training labels as the data collection and annotation procedure is extremely expensive. As a remedy, leveraging the large-scale images is an attractive solution [3]. However, in doing so, we face three main issues: 1) lacking video supervision in images, 2) semantic label inconsistency between images and videos, and 3) the domain gap (e.g., explicit data styles or implicit data distributions are different) between images and videos. The current learning paradigm bypasses the first two issues by independently training the detection head and tracking head with images and videos (decoupled). Instead, our learning framework explicitly handles the former two issues by hallucinating the supervisions and enables end-to-end video model learning from all training data, leading