A Study on Evaluation on Opinion Retrieval Systems

We study the evaluation of opinion retrieval systems. Opinion retrieval is a relatively new research area, nevertheless classical evaluation measures, those adopted for ad hoc retrieval, such as MAP, precision at 10 etc., were used to assess the quality of rankings. In this paper we investigate the effectiveness of these standard evaluation measures for topical opinion retrieval. In doing this we split the opinion dimension from the relevance one and use opinion classiers, with varying accuracy, to analyse how opinion retrieval performance changes by perturbing the outcomes of the opinion classiers. Classiers could be studied in two modalities, that is either to re-rank or to lter out directly documents obtained through a rst relevance retrieval. In this paper we formally outline both approaches, while for now focussing on the ltering process. The proposed approach aims to establish the correlation between the accuracy of the classiers and the performance of the topical opinion retrieval. In this way it will be possible to assess the eectiveness of the opinion component by comparing the eectiveness of the relevance baseline with that of the topical opinion.