Incorporating Efficiency in Evaluation

The trade-o between retrieval eciency and eectiveness naturally arises in various domains of information retrieval research. In this paper, we argue that the users’ reaction to the system eciency should be incorporated into the cascade model of the user behaviour and that an eectiveness-awar e modication of a state-of-the-art evaluation metric can be derived. We propose to base the study on the anonymised browser/toolbar log data, and our preliminary experiments demonstrate that the users’ reaction to the delays observed in the dataset is consistent with one reported previously.