Combining Evidence for Relevance Criteria: A Framework and Experiments in Web Retrieval

We present a framework that assesses relevance with respect to several relevance criteria, by combining the query-dependent and query-independent evidence indicating these criteria. This combination of evidence is modelled in a uniform way, irrespective of whether the evidence is associated with a single document or related documents. The framework is formally expressed within Dempster-Shafer theory. It is evaluated for web retrieval in the context of TREC’s Topic Distillation task. Our results indicate that aggregating content-based evidence from the linked pages of a page is beneficial, and that the additional incorporation of their homepage evidence further improves the effectiveness.

[1]  David Hawking,et al.  Overview of the TREC 2003 Web Track , 2003, TREC.

[2]  Djoerd Hiemstra,et al.  The Importance of Prior Probabilities for Entry Page Search , 2002, SIGIR '02.

[3]  Glenn Shafer,et al.  A Mathematical Theory of Evidence , 2020, A Mathematical Theory of Evidence.

[4]  Carol L. Barry.  User-Defined Relevance Criteria: An Exploratory Study , 1994, J. Am. Soc. Inf. Sci..

[5]  David Hawking,et al.  Overview of the TREC 2004 Web Track , 2004, TREC.

[6]  Fabio Crestani,et al.  Lectures on Information Retrieval , 2001, Lecture Notes in Computer Science.

[7]  Iadh Ounis,et al.  Usefulness of hyperlink structure for query-biased topic distillation , 2004, SIGIR '04.

[8]  Kevin S. McCurley,et al.  Untangling compound documents on the web , 2003, HYPERTEXT '03.

[9]  Mounia Lalmas,et al.  Advances in XML Information Retrieval, Third International Workshop of the Initiative for the Evaluation of XML Retrieval, INEX 2004, Dagstuhl Castle, Germany, December 6-8, 2004, Revised Selected Papers , 2005, INEX.

[10]  Tao Qin,et al.  A study of relevance propagation for web search , 2005, SIGIR '05.

[11]  Yves Chiaramella,et al.  Information Retrieval and Structured Documents , 2000, ESSIR.

[12]  Arthur P. Dempster,et al.  A Generalization of Bayesian Inference , 1968, Classic Works of the Dempster-Shafer Theory of Belief Functions.

[13]  Gabriella Kazai,et al.  The Accessibility Dimension for Structured Document Retrieval , 2002, ECIR.

[14]  W. Bruce Croft,et al.  The INQUERY Retrieval System , 1992, DEXA.

[15]  Stephen E. Robertson,et al.  Relevance weighting for query independent evidence , 2005, SIGIR '05.

[16]  Ian Soboroff.  On evaluating web search with very few relevant documents , 2004, SIGIR '04.

[17]  Sergey Brin,et al.  The Anatomy of a Large-Scale Hypertextual Web Search Engine , 1998, Comput. Networks.

[18]  Massimo Marchiori,et al.  The Quest for Correct Information on the Web: Hyper Search Engines , 1997, Comput. Networks.