Evaluation and User Preference Study on Spatial Diversity

Spatial diversity is a relatively new branch of research in the context of spatial information retrieval. Although the assumption that spatially diversified results may meet users’ needs better seems reasonable, there has been little hard evidence in the literature indicating so. In this paper, we will show the potentials of spatial diversity by not only the traditional evaluation metrics (precision and cluster recall), but also through a user preference study using Amazon Mechanical Turk. The encouraging results from the latter prove that users do have strong preference on spatially diversified results.

[1]  Avi Arampatzis,et al.  Distributed Ranking Methods for Geographic Information Retrieval , 2004, SDH.

[2]  Hermann Ney,et al.  Jointly optimising relevance and diversity in image retrieval , 2009, CIVR '09.

[3]  Hua Li,et al.  Improving web search results using affinity graph , 2005, SIGIR '05.

[4]  Panagiotis G. Ipeirotis,et al.  Automatic Extraction of Useful Facet Hierarchies from Text Databases , 2008, 2008 IEEE 24th International Conference on Data Engineering.

[5]  Paul Clough,et al.  The IAPR TC-12 Benchmark: A New Evaluation Resource for Visual Information Systems , 2006 .

[6]  Jiayu Tang,et al.  Generic and Spatial Approaches to Image Search Results Diversification , 2009, ECIR.

[7]  Jade Goldstein-Stewart,et al.  The use of MMR, diversity-based reranking for reordering documents and producing summaries , 1998, SIGIR '98.

[8]  Stephen E. Robertson,et al.  On GMAP: and other transformations , 2006, CIKM '06.

[9]  Omar Alonso,et al.  Crowdsourcing for relevance evaluation , 2008, SIGF.

[10]  David A. Forsyth,et al.  Utility data annotation with Amazon Mechanical Turk , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[11]  Charles L. A. Clarke,et al.  Novelty and diversity in information retrieval evaluation , 2008, SIGIR '08.

[12]  Paul Clough,et al.  Creating a test collection to evaluate diversity in image retrieval , 2008, SIGIR 2008.

[13]  Peter Ingwersen,et al.  Developing a Test Collection for the Evaluation of Integrated Search , 2010, ECIR.

[14]  Hideo Joho,et al.  Judging the Spatial Relevance of Documents for GIR , 2006, ECIR.