A Formal Approach to Effectiveness Metrics for Information Access: Retrieval, Filtering, and Clustering

In this tutorial we present a formal account of evaluation metrics for three of the most salient information related tasks: Retrieval, Clustering, and Filtering. We focus on the most popular metrics and, by exploiting measurement theory, we show some constraints for suitable metrics in each of the three tasks. We also systematically compare metrics according to how they satisfy such constraints, we provide criteria to select the most adequate metric for each specific information access task, and we discuss how to combine and weight metrics.

[1]  Ben Carterette,et al.  System effectiveness, user models, and user utility: a conceptual framework for investigation , 2011, SIGIR.

[2]  Byron Dom,et al.  An Information-Theoretic External Cluster-Validity Measure , 2002, UAI.

[3]  Charles L. A. Clarke,et al.  Time-based calibration of effectiveness measures , 2012, SIGIR '12.

[4]  Peter Ingwersen,et al.  Developing a Test Collection for the Evaluation of Integrated Search , 2010, ECIR.

[5]  Julio Gonzalo,et al.  Combining Evaluation Metrics via the Unanimous Improvement Ratio and its Application to Clustering Tasks , 2011, J. Artif. Intell. Res..

[6]  Julio Gonzalo,et al.  A comparison of extrinsic clustering evaluation metrics based on formal constraints , 2008, Information Retrieval.

[7]  Julio Gonzalo,et al.  A general evaluation measure for document organization tasks , 2013, SIGIR.

[8]  Eddy Maddalena,et al.  Axiometrics: Axioms of Information Retrieval Effectiveness Metrics , 2014, EVIA@NTCIR.

[9]  Alistair Moffat,et al.  Rank-biased precision for measurement of retrieval effectiveness , 2008, TOIS.

[10]  Stefano Mizzaro,et al.  A Classification of IR Effectiveness Metrics , 2006, ECIR.

[11]  Stefano Mizzaro,et al.  Axiometrics: An Axiomatic Approach to Information Retrieval Effectiveness Metrics , 2013, ICTIR.

[12]  Eddy Maddalena,et al.  The Axiometrics Project , 2014, IIR.

[13]  Julio Gonzalo,et al.  A general account of effectiveness metrics for information tasks: retrieval, filtering, and clustering , 2014, SIGIR.