Evaluation of personal search over an individual's personal information space on the desktop or elsewhere is problematic for reasons relating both to the personal and private nature of the data and the associated personal information needs of collection owners. Indeed challenges associated with evaluation in this space are recognised as one of the key factors hindering the development of research in personal information retrieval. We present the "personal information retrieval evaluation (PIRE)" tool, which provides a solution to this evaluation problem using a ‘living laboratory' approach. This tool allows for the evaluation of retrieval techniques using ‘real' individuals' personal collections, queries and result sets, in a cross-comparable repeatable way, while importantly maintaining an individual's informational privacy.
[1]
Hinrich Schütze,et al.
Introduction to information retrieval
,
2008
.
[2]
Craig A. N. Soules,et al.
Connections: using context to enhance file search
,
2005,
SOSP '05.
[3]
Susan T. Dumais,et al.
Evaluation Challenges and Directions for Information-Seeking Support Systems
,
2009,
Computer.
[4]
W. Bruce Croft,et al.
Retrieval experiments using pseudo-desktop collections
,
2009,
CIKM.
[5]
Emanuele Della Valle,et al.
An Introduction to Information Retrieval
,
2013
.
[6]
Christopher D. Manning,et al.
Introduction to Information Retrieval
,
2010,
J. Assoc. Inf. Sci. Technol..