Statistical Modeling of Diversity in Top-N Recommender Systems

This paper provides the first statistical analysis of recommendation diversity. We propose a model that allows diversity to be evaluated analytically using the concentration index, a statistical measure of diversity. While this model abstracts the recommendation process, it provides good overall agreement with real recommendation algorithms. Using the model we are able to analytically demonstrate the trade-off between diversity and overall system performance. Moreover, the model makes explicit the various choices that are available to the algorithm designer to improve recommendation diversity. Our exploration of these choices provides good insight into what can be achieved in practice by algorithms that attempt to provide greater recommendation diversity without significant degradation of system precision.