Evaluating Personal Information Management Using an Activity Logs Enriched Desktop Dataset

The effective evaluation of Personal Information Management is a crucial problem for the research community. While evaluation methodologies for retrieval on the Web and in digital libraries are well-developed, the experiments with the advanced desktop tools are neither repeatable nor comparable. As privacy concerns do not allow to copy and distribute personal data outside the research lab, we suggest to overcome this problem by creation of desktop datasets within different research groups using a single methodology and a common set of tools. A dataset can include not only a static snapshot of the desktop documents, but also the logs of user activity on the desktop within several last months. We present the structure of the required dataset, a set of implemented tools and a sample dataset collected within the L3S Research Center. ACM Classification

[1]  David Hawking,et al.  Workload sampling for enterprise search evaluation , 2007, SIGIR.

[2]  Mark Claypool,et al.  Implicit interest indicators , 2001, IUI '01.

[3]  Ravi Kumar,et al.  Searching with context , 2006, WWW '06.

[4]  Cyril Cleverdon,et al.  The Cranfield tests on index language devices , 1997 .

[5]  Mert Bay,et al.  A "do-it-yourself" evaluation service for music information retrieval systems , 2007, SIGIR.

[6]  A. Poggi,et al.  Intelligent user task oriented systems , 2006 .

[7]  Peter Ingwersen,et al.  The development of a method for the evaluation of interactive information retrieval systems , 1997, J. Documentation.

[8]  Ellen M. Voorhees,et al.  The Philosophy of Information Retrieval Evaluation , 2001, CLEF.

[9]  Wolfgang Nejdl,et al.  Semantically Enhanced Searching and Ranking on the Desktop , 2005, Semantic Desktop Workshop.

[10]  Susan T. Dumais,et al.  Learning user interaction models for predicting web search result preferences , 2006, SIGIR.

[11]  Wolfgang Nejdl,et al.  Building a Desktop Search Test-Bed , 2007, ECIR.

[12]  Wolfgang Nejdl,et al.  Desktop Context Detection Using Implicit Feedback , 2006 .

[13]  Gerard Salton,et al.  Research and Development in Information Retrieval , 1982, Lecture Notes in Computer Science.

[14]  Feng Qiu,et al.  Automatic identification of user interest for personalized search , 2006, WWW '06.

[15]  Beverly Yang,et al.  Retroactive answering of search queries , 2006, WWW '06.

[16]  Susan T. Dumais,et al.  Personalizing Search via Automated Analysis of Interests and Activities , 2005, SIGIR.

[17]  David Elsweiler,et al.  Towards task-based personal information management evaluations , 2007, SIGIR.