Monitoring User-System Performance in Interactive Retrieval Tasks

Monitoring user-system performance in interactive search is a challenging task. Traditional measures of retrieval evaluation, based on recall and precision, are not of any use in real time, for they require a priori knowledge of relevant documents. This paper shows how a Shannon entropy-based measure of user-system performance naturally falls in the framework of (interactive) probabilistic information retrieval. The value of entropy of the distribution of probability of relevance associated with the documents in the collection can be used to monitor search progress in live testing, to allow for example the system to select an optimal combination of search strategies. User profiling and tuning parameters of retrieval systems are other important applications.

[1]  Djoerd Hiemstra,et al.  Relevance feedback in probabilistic multimedia retrieval , 2003 .

[2]  Ben Shneiderman,et al.  Designing the User Interface: Strategies for Effective Human-Computer Interaction , 1998 .

[3]  Michael E. Lesk,et al.  Relevance assessments and retrieval system evaluation , 1968, Inf. Storage Retr..

[4]  Ellen M. Voorhees.  Variations in relevance judgments and the measurement of retrieval effectiveness , 2000, Inf. Process. Manag..

[5]  W. Bruce Croft,et al.  Using Probabilistic Models of Document Retrieval without Relevance Information , 1979, J. Documentation.

[6]  Ingemar J. Cox,et al.  The Bayesian image retrieval system, PicHunter: theory, implementation, and psychophysical experiments , 2000, IEEE Trans. Image Process..

[7]  Djoerd Hiemstra,et al.  A Probabilistic Multimedia Retrieval Model and Its Evaluation , 2003, EURASIP J. Adv. Signal Process..

[8]  Ellen M. Voorhees,et al.  The eleventh text REtrieval conference, TREC 2002 , 2003 .

[9]  Ben Shneiderman,et al.  Designing the user interface - strategies for effective human-computer interaction, 3rd Edition , 1997 .

[10]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[11]  Stephen E. Robertson,et al.  Probabilistic models of indexing and searching , 1980, SIGIR '80.

[12]  S. Robertson.  The probability ranking principle in IR , 1997 .

[13]  Warren R. Greiff,et al.  The maximum entropy approach and probabilistic IR models , 2000, TOIS.

[14]  Stephen E. Robertson,et al.  Relevance weighting of search terms , 1976, J. Am. Soc. Inf. Sci..

[15]  Nuno Vasconcelos,et al.  A probabilistic architecture for content-based image retrieval , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[16]  William S. Cooper,et al.  Exploiting the maximum entropy principle to increase retrieval effectiveness , 1983, J. Am. Soc. Inf. Sci..

[17]  Ellen M. Voorhees,et al.  The Twelfth Text Retrieval Conference, TREC 2003 , 2004 .