Querying Video Data by Spatio-Temporal Relationships of Moving Object Traces

We propose a query method that allows viewers to search video data based on the spatio-temporal relationships of moving objects using a simple and intuitive mode of input. In order to make a query, for example, of a certain segment or style of play in sports footage, it is necessary to express the complex movements of objects in an intuitive manner. In the proposed method, the user inputs this information as an object trajectory using a mouse or pen via a GUI. The usability of our proposed method is evaluated using a prototype query system with four main function; video query based on object speed, spatially absolute or relative query based on object movement, querying of movements of multiple moving objects specifying the spatio-temporal relationships among objects by expressing each object’s trace on a timeline or multi-canvas, and providing a non-linear viewing experience by allowing the user to control playback by drawing object traces directly on the video screen while watching the video.