Diversity-based interactive learning meets multimodality

Abstract In interactive retrieval tasks, one of the main objectives is to maximize the user information gain throughout search sessions. Retrieving many relevant items is quite important, but it does not necessarily completely satisfy the user needs. When only relevant near-duplicate items are retrieved, the amount of different concepts users are able to extract from the target collection is very limited. Therefore, broadening the number of concepts present in a result set may improve the overall search experience. Diversifying concepts present in the retrieved set is one possibility for increasing the information gain in a single search iteration, maximizing the likelihood of including at least some relevant items for each possible intent of ambiguous or underspecified queries. Relevance feedback approaches may also take advantage of diverse results to improve internal machine learning models. In this context, this work proposes and analyses several multimodal image retrieval approaches built over a learning framework for relevance feedback on diversified results. Our experimental analysis shows that different retrieval modalities are positively impacted by diversity, but achieve best retrieval effectiveness with diversification applied at different moments of a search session. Moreover, the best results are achieved with a query-by-example approach using multimodal information obtained from feedback. In summary, we demonstrate that learning with diversity is an effective alternative for boosting multimodal interactive learning approaches.