Report on the SIGIR 2009 workshop on the future of IR evaluation

On July 23, 2009 the SIGIR Workshop on the Future of IR Evaluation was held as part of SIGIR in Boston. The program consisted of four keynotes, a boaster and poster session with 20 accepted papers, four breakout groups, and a final panel discussion of the breakout group reports. This report outlines the events of the workshop and summarizes the major outcomes.

[1]  Miguel Costa,et al.  Towards Information Retrieval Evaluation over Web Archives , 2009 .

[2]  Neal Kiritkumar Lathia,et al.  Evaluating collaborative filtering over time , 2009, SIGIR 2009.

[3]  Gabriella Kazai,et al.  On the Evaluation of the Quality of Relevance Assessments Collected through Crowdsourcing , 2009 .

[4]  Allan Hanbury,et al.  Toward Automated Component-Level Evaluation , 2009 .

[5]  Ralf Schenkel,et al.  Evaluating Network-aware Retrieval in Social Networks , 2009, SIGIR 2009.

[6]  Cyril W. Cleverdon,et al.  Report on the first stage of an investigation into the comparative efficiency of indexing systems , 1960 .

[7]  Cecile Paris,et al.  Stakeholders and their respective costs-benefits in IR evaluation , 2009 .

[8]  Yiming Yang,et al.  CiteEval for Evaluating Personalized Social Web Search , 2009 .

[9]  James Allan,et al.  Meeting of the MINDS: an information retrieval research agenda , 2007, SIGF.

[10]  Alistair Moffat,et al.  EvaluatIR: an online tool for evaluating and comparing IR systems , 2009, SIGIR.

[11]  Carol Peters,et al.  Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation , 2009 .

[12]  Milad Shokouhi,et al.  Are Evaluation Metrics Identical With Binary Judgements ? , 2009 .

[13]  Sofia Stamou,et al.  Queries without Clicks: Successful or Failed Searches? , 2009 .

[14]  Alistair Moffat,et al.  Relative significance is insufficient: Baselines matter too , 2009, SIGIR 2009.

[15]  Mariano P. Consens,et al.  Enhanced Web Retrieval Task , 2009 .

[16]  David Hawking,et al.  New methods for creating testfiles: Tuning enterprise search with C-TEST , 2009, SIGIR 2009.

[17]  Susan T. Dumais,et al.  Evaluation Challenges and Directions for Information-Seeking Support Systems , 2009, Computer.

[18]  Karen Spärck Jones.  What's the value of TREC: is there a gap to jump or a chasm to bridge? , 2006, SIGF.

[19]  and software — performance evaluation , .

[20]  Fernando Llopis,et al.  How long can you wait for your QA system , 2009 .

[21]  W. Redmond,et al.  Accounting for Stability of Retrieval Algorithms using Risk-Reward Curves , 2009 .

[22]  Ji-Rong Wen,et al.  Building a Test Collection for Evaluating Search Result Diversity : A Preliminary Study , 2009 .

[23]  Mark D. Smucker,et al.  A Plan for Making Information Retrieval Evaluation Synonymous with Human Performance Prediction , 2009 .

[24]  Nicholas J. Belkin,et al.  A Model for Evaluation of Interactive Information Retrieval , 2009 .