A comparison of search term weighting: term relevance vs. inverse document frequency

The term relevance weighting method has been shown to produce optimal information retrieval queries under well-defined conditions. The parameters needed to generate the term relevance factors cannot unfortunately be estimated accurately in practice; futhermore, in realistic test situations, it appears difficult to obtain improved retrieval results using the term relevance weights over much simpler term weighting systems such as, for example, the inverse document frequency weights.It is shown in this study that the inverse document frequency weights and the term relevance weights are closely related over a wide range of the frequency spectrum. Methods are introduced for estimating the term relevance weights, and experimental results are given comparing the inverse document frequency with the estimated term relevance weights.

[1]  Clement T. Yu,et al.  Term Weighting in Information Retrieval Using the Term Precision Model , 1982, JACM.

[2]  Karen Spärck Jones.  Search Term Relevance Weighting given Little Relevance Information , 1997, J. Documentation.

[3]  Karen Spärck Jones.  Experiments in relevance weighting of search terms , 1979, Inf. Process. Manag..

[4]  Stephen E. Robertson,et al.  Relevance weighting of search terms , 1976, J. Am. Soc. Inf. Sci..

[5]  W. Bruce Croft,et al.  Using Probabilistic Models of Document Retrieval without Relevance Information , 1979, J. Documentation.

[6]  R. K. Waldstein,et al.  Term relevance weights in on-line information retrieval , 1977, Inf. Process. Manag..

[7]  Clement T. Yu,et al.  Precision Weighting—An Effective Automatic Indexing Method , 1976, J. ACM.

[8]  Clement T. Yu,et al.  Automatic indexing using term discrimination and term precision measurements , 1976, Information Processing & Management.

[9]  Stephen E. Robertson,et al.  Probabilistic models of indexing and searching , 1980, SIGIR '80.

[10]  C. J. van Rijsbergen,et al.  An Evaluation of feedback in Document Retrieval using Co‐Occurrence Data , 1978, J. Documentation.

[11]  Karen Sparck Jones.  A statistical interpretation of term specificity and its application in retrieval , 1972 .

[12]  Karen Sparck Jones.  Search term relevance weighting- some recent results , 1979 .

[13]  Donald H. Kraft,et al.  Evaluation of information retrieval systems: A decision theory approach , 1978, J. Am. Soc. Inf. Sci..

[14]  Clement T. Yu,et al.  On models of information retrieval processes , 1979, Inf. Syst..