Overview of the TREC 2014 Federated Web Search Track

The TREC Federated Web Search track is intended to promote research related to federated search in a realistic web setting, and hereto provides a large data collection gathered from a series of online search engines. This overview paper discusses the results of the first edition of the track, FedWeb 2013. The focus was on basic challenges in federated search: (1) resource selection, and (2) results merging. After an overview of the provided data collection and the relevance judgments for the test topics, the participants’ individual approaches and results on both tasks are discussed. Promising research directions and an outlook on the 2014 edition of the track are provided as well.

[1]  Fernando Diaz,et al.  Sources of evidence for vertical selection , 2009, SIGIR.

[2]  Djoerd Hiemstra,et al.  Mirex and Taily at TREC 2013 , 2013, TREC.

[3]  Wang Banyue,et al.  Chapter 5 , 2003 .

[4]  Milad Shokouhi,et al.  Federated Search , 2011, Found. Trends Inf. Retr..

[5]  Xiaohua Hu,et al.  Drexel at TREC 2014 Federated Web Search Track , 2014, TREC.

[6]  B. Southam.  Query , 1902, Canadian Medical Association Journal.

[7]  Djoerd Hiemstra,et al.  Ranking XPaths for extracting search result records , 2012 .

[8]  Mandar Mitra,et al.  ISI at the TREC 2013 Federated task , 2013, TREC.

[9]  Ying Li,et al.  KDD CUP-2005 report: facing a great challenge , 2005, SKDD.

[10]  João Magalhães,et al.  NovaSearch at TREC 2013 Federated Web Search Track: Experiments with rank fusion , 2013, TREC.

[11]  Djoerd Hiemstra,et al.  Exploiting user disagreement for web search evaluation: an experimental approach , 2014, WSDM.

[12]  Jimmy J. Lin,et al.  CWI and TU Delft at TREC 2013: Contextual Suggestion, Federated Web Search, KBA, and Web Tracks , 2013 .

[13]  Andrei Z. Broder,et al.  Estimating corpus size via queries , 2006, CIKM '06.

[14]  Tetsuya Sakai,et al.  On the reliability and intuitiveness of aggregated search metrics , 2013, CIKM.

[15]  Fernando Diaz,et al.  A Methodology for Evaluating Aggregated Search Results , 2011, ECIR.

[16]  Dong Nguyen,et al.  Overview of the TREC 2013 Federated Web Search Track (draft) , 2013 .

[17]  Djoerd Hiemstra,et al.  Federated search in the wild: the combined power of over a hundred search engines , 2012, CIKM '12.

[18]  James P. Callan,et al.  Query Transformations for Result Merging , 2014, TREC.

[19]  Htay New Daw.  Does my child have ADHD , 2007 .

[20]  Yue Liu,et al.  ICTNET at Federated Web Search Track 2014 , 2014, TREC.

[21]  Ben Carterette,et al.  University of Delaware at TREC 2014 , 2014, TREC.

[22]  Craig Willis,et al.  The University of Illinois' Graduate School of Library and Information Science at TREC 2011 , 2011, TREC.

[23]  James H. Martin,et al.  Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition , 2000 .

[24]  Wei Cao,et al.  RUC at TREC 2014: Select Resources Using Topic Models , 2014, TREC.

[25]  Jamie Callan,et al.  DISTRIBUTED INFORMATION RETRIEVAL , 2002 .

[26]  Jaana Kekäläinen,et al.  Using graded relevance assessments in IR evaluation , 2002, J. Assoc. Inf. Sci. Technol..

[27]  Sergey Brin,et al.  The Anatomy of a Large-Scale Hypertextual Web Search Engine , 1998, Comput. Networks.

[28]  Krisztian Balog.  NTNUiS at the TREC 2014 Federated Web Search Track , 2014 .

[29]  Djoerd Hiemstra,et al.  Aligning Vertical Collection Relevance with User Intent , 2014, CIKM.

[30]  Gregory N. Hullender,et al.  Learning to rank using gradient descent , 2005, ICML.

[31]  Djoerd Hiemstra,et al.  SearchResultFinder: federated search made easy , 2013, SIGIR.

[32]  Massimo Melucci,et al.  University of Padua at TREC 2014: Federated Web Search Track , 2014, TREC.

[33]  Fabio Crestani,et al.  Opinions in Federated Search: University of Lugano at TREC 2014 Federated Web Search Track , 2014, TREC.

[34]  Olivier Chapelle,et al.  Expected reciprocal rank for graded relevance , 2009, CIKM.

[35]  David Hawking,et al.  Server selection methods in hybrid portal search , 2005, SIGIR '05.

[36]  Moses Charikar,et al.  Similarity estimation techniques from rounding algorithms , 2002, STOC '02.

[37]  Thomas Demeester,et al.  What Snippets Say about Pages in Federated Web Search , 2012, AIRS.

[38]  Djoerd Hiemstra,et al.  Two selfless contributions to web search evaluation , 2014, TREC.

[39]  Krisztian Balog.  Collection and Document Language Models for Resource Selection , 2013, TREC.

[40]  Man Lan,et al.  Simple May Be Best - A Simple and Effective Method for Federated Web Search via Search Engine Impact Factor Estimation , 2014, TREC.