Semantics reinforcement and fusion learning for multimedia streams

Fusion of multimedia streams for enhanced performance is a critical problem for retrieval. However, fusion performance tends to easily overfit the hillclimb set used to learn fusion rules. In this paper, we perform fusion learning for multimedia streams using a greedy performance driven algorithm. In our fusion learning paradigm, fused output is a linear combination of multiple classifiers or ranked streams. The algorithm is inspired from Ensemble Learning [2] but takes that idea further for improving generalization capability. A key application of our fusion learning algorithm, described in this work, is semantics reinforcement using an ensemble of classifiers built using the same training dataset but groundtruth corresponding to different concepts. We expect that classifiers built for semantically close concepts should reinforce each other's performance and fusion learning is an excellent post-classification way to reinforce semantics and performance. Fusion learning experiments have been performed on TRECVID 2005 test set. Experiments using the well established retrieval effectiveness measure of mean average precision reveal that our proposed algorithm improves over the best classifier (oracle) by 3.8%. We also present and discuss some interesting and intuitive semantic reinforcement trends observed during fusion learning.