Modified rough set based aggregation for effective evaluation of web search systems

Rank Aggregation is the problem of generating a single consensus ranking for a given set of rankings. Rough set based Rank aggregation is a user feedback based technique for rank aggregation, which learns ranking rules using rough set theory. In this paper, we discuss an improved version of the Rough set based Rank aggregation technique, which is more suitable for aggregation of different Web search evaluation techniques. For learning the ranking rules, we obtain the implicit user feedback to the search results returned by a search engine in response to a set of fifteen queries and mine the ranking rules using rough set theory. In the modified rough set based rank aggregation technique, we incorporate the confidence of the rules in predicting a class for a given set of data. That means, we do not say surely that the record belongs to a particular class according to a particular rule. Instead, we associate a score variable to the predicted class of the record, where the value of the variable is equal to the confidence measure of the rule. We validate the mined ranking rules by comparing the predicted user feedback based ranking with the actual user feedback based ranking. We apply the ranking rules to another set of thirty seven queries for aggregating different rankings of search results obtained on the basis of different evaluation techniques. We show our experimental results pertaining to seven public search engines.