Contextual Person Identification in Multimedia Data

Automatic tracking and identification of faces and persons are essential tasks in many video analysis systems, for example to automatically generate meta data or as basis for higher level applications. In many cases, identification is based on a single modality such as faces. In this work, we propose methods to improve person identification by integration of multiple cues including multiple modalities and contextual information. We motivate and evaluate our proposed methods in the context of multimedia data, specifically TV series. Despite its usually high resolution, multimedia data presents many challenges. For example, camera views change constantly at shot boundaries, the camera position is generally unknown and image conditions and poses of faces and poses can change rapidly due to the underlying plot. Since we make only few assumptions about the underlying data, our methods are applicable to other domains as well, for example in the area of safety and security. Before we can identify a face, it has to be localized in the image first. In videos we can further associate localizations over time to consecutive face tracks. Face tracks can then be identified jointly and errors in single frames (e.g., due to noise in the data or imprecise localization) can be mitigated, improving overall identification accuracy. In this work, we propose a detector-based face tracking approach based on a large bank of detectors which cover a range of head poses. We integrate the detectors in a particle filter such that these can be used efficiently, i.e. only one detector out of 49 is evaluated for each particle. We evaluate our approach on a data set of two TV series, which we annotated with ground truth face positions. The data set contains over 100 000 annotated faces and is one of the largest public data sets available for the evaluation of face tracking. With our proposed tracking approach we achieve an improvement of 0.15 in Multiple Object