Aggregating Subjective and Objective Measures of Web Search Quality using Modified Shimura Technique

Web searching is perhaps the second most popular activity on Internet. Millions of users search the Web daily for their purpose. But as there are a number of search engines available, there must be some procedure to evaluate them. In this paper, we try to present an effort in this regard. For subjective measure, we are taking into account the "satisfaction " a user gets when presented with search results. The feedback of the user is inferred from watching the actions of the user on the search results presented before him in response to his query. For objective measures, we use Vector space model and Boolean similarity measures. All the three measures are aggregated using modified Shimura technique of rank aggregation. The aggregated ranking is then compared with the original ranking given by the search engine. The correlation coefficient thus obtained is averaged for a set of queries. We show our experimental results pertaining to seven public search engines and fifteen queries.

[1]  Rashid Ali,et al.  A subjective cum objective measure of Web search quality , 2005, Proceedings of 2005 International Conference on Intelligent Sensing and Information Processing, 2005..

[2]  Andrei Z. Broder,et al.  A Technique for Measuring the Relative Size and Overlap of Public Web Search Engines , 1998, Comput. Networks.

[3]  Michael McGill,et al.  Introduction to Modern Information Retrieval , 1983 .

[4]  Longzhuang Li,et al.  Precision Evaluation of Search Engines , 2004, World Wide Web.

[5]  Longzhuang Li,et al.  A new method for automatic performance comparison of search engines , 2004, World Wide Web.

[6]  M. F. Porter,et al.  An algorithm for suffix stripping , 1997 .

[7]  Marc Najork,et al.  Measuring Index Quality Using Random Walks on the Web , 1999, Comput. Networks.

[8]  Ronald R. Yager,et al.  On ordered weighted averaging aggregation operators in multicriteria decisionmaking , 1988, IEEE Trans. Syst. Man Cybern..

[9]  Giles,et al.  Searching the world wide Web , 1998, Science.

[10]  C. Lee Giles,et al.  Accessibility of information on the web , 1999, Nature.

[11]  Peter B. Danzig,et al.  Boolean Similarity Measures for Resource Discovery , 1997, IEEE Trans. Knowl. Data Eng..

[12]  Ronald R. Yager,et al.  On ordered weighted averaging aggregation operators in multicriteria decision-making , 1988 .

[13]  Steve Chien,et al.  Approximating Aggregate Queries about Web Pages via Random Walks , 2000, VLDB.

[14]  Donna K. Harman,et al.  Results and Challenges in Web Search Evaluation , 1999, Comput. Networks.

[15]  Marc Najork,et al.  On near-uniform URL sampling , 2000, Comput. Networks.

[16]  M. Shimura.  Fuzzy sets concept in rank-ordering objects , 1973 .

[17]  W. Pirie.  Spearman Rank Correlation Coefficient , 2006 .