Community Question Answering (CQA) platforms contain a large number of questions and associated answers. Answerers sometimes include URLs as part of the answers to provide further information. This paper describes a novel way of building a test collection for web search by exploiting the link information from this type of social media data. We propose to build the test collection by regarding CQA questions as queries and the associated linked web pages as relevant documents. To evaluate this approach, we collect approximately ten thousand CQA queries, whose answers contained links to ClueWeb09 documents after spam filtering. Experimental results using this collection show that the relative effectiveness between different retrieval models on the ClueWeb-CQA query set is consistent with that on the TREC Web Track query sets, confirming the reliability of our test collection. Further analysis shows that the large number of queries generated through this approach compensates for the sparse relevance judgments in determining significant differences.
[1]
W. Bruce Croft,et al.
Evaluating verbose query processing techniques
,
2010,
SIGIR.
[2]
James Allan,et al.
Minimal test collections for retrieval evaluation
,
2006,
SIGIR.
[3]
W. Bruce Croft,et al.
A Markov random field model for term dependencies
,
2005,
SIGIR '05.
[4]
Charles L. A. Clarke,et al.
Efficient and effective spam filtering and re-ranking for large web datasets
,
2010,
Information Retrieval.
[5]
Ellen M. Voorhees,et al.
Retrieval evaluation with incomplete information
,
2004,
SIGIR '04.
[6]
W. Bruce Croft,et al.
Latent concept expansion using markov random fields
,
2007,
SIGIR.
[7]
James Allan,et al.
Evaluation over thousands of queries
,
2008,
SIGIR '08.
[8]
Charles L. A. Clarke,et al.
Reliable information retrieval evaluation with incomplete and biased judgements
,
2007,
SIGIR.
[9]
C. J. van Rijsbergen,et al.
Report on the need for and provision of an 'ideal' information retrieval test collection
,
1975
.