论文信息 - New Measures for Open-Domain Question Answering Evaluation Within a Time Constraint

New Measures for Open-Domain Question Answering Evaluation Within a Time Constraint

Previous works on evaluating the performance of Question Answering (QA) systems are focused on the evaluation of the precision. In this paper, we developed a mathematic procedure in order to explore new evaluation measures in QA systems considering the answer time. Also, we carried out an exercise for the evaluation of QA systems within a time constraint in the CLEF-2006 campaign, using the proposed measures. The main conclusion is that the evaluation of QA systems in realtime can be a new scenario for the evaluation of QA systems.

[1] Ellen M. Voorhees,et al. The TREC 2005 robust track , 2006, SIGF.

[2] Valentin Jijkoun,et al. Overview of the CLEF 2006 Multilingual Question Answering Track , 2006, CLEF.

[3] Tsuneaki Kato,et al. An Overview of Question and Answering Challenge (QAC) of the Next NTCIR Workshop , 2001, NTCIR.

[4] Maarten de Rijke,et al. The Multilingual Question Answering Track at CLEF , 2006, LREC.

[5] 福本淳一,et al. Question answering challenge (QAC1) , 2002 .

[6] Ellen M. Voorhees,et al. Overview of the TREC 2004 Novelty Track. , 2005 .