An Overview of NTCIR-5 QAC3

This paper provides an overview of NTCIR-5 QAC3 (Question Answering Challenge 3). QAC is a series of challenges for evaluating question answering technologies in Japanese. QAC3 follows the same course as QAC based on the success of the previous two workshops, with its task limited to that corresponding to QAC2 Subtask 3 aiming at the convergence of research resources for novel subjects. This task assumes interactive use of QA systems and evaluates, among other things, the abilities needed under such circumstances, i.e. proper interpretation of questions under a given dialogue context; in other words, context processing abilities such as anaphora resolution and ellipses handling (hereafter we refer to the task as the IAD task, where IAD stands for Information Access Dialogue, and to the whole workshop as QAC3). The IAD task in QAC3 is based on QAC2 Subtask 3 with several improvements, including elaboration of the scope of questions and answers and introduction of multi-grade evaluation and the concept of a correct answer set. In addition, a new WoZ method was devised and applied in the QAC3 test set construction. QAC3 had as many participants as QAC2 Subtask 3, and new trials and advances in existing methods were observed from the submission results.

[1]  Katunobu Itou,et al.  Question Answering Experiments at NTCIR-5: Acquisition of Answer Evaluation Patterns and Context Processing using Passage Retrieval , 2005, NTCIR.

[2]  Tsuneaki Kato,et al.  Question Answering Challenge (QAC-1): An Evaluation of Question Answering Tasks at the NTCIR Workshop 3 , 2003, New Directions in Question Answering.

[3]  Jun-ichi Fukumoto,et al.  Answering Questions of IAD Task using Reference Resolution of Follow-up Questions , 2005, NTCIR.

[4]  Ellen M. Voorhees,et al.  Building a question answering test collection , 2000, SIGIR '00.

[5]  Tatsunori Mori,et al.  Answering Contextual Questions Based on the Cohesion with the Knowledge - Yokohama National University at NTCIR-5 QAC3 , 2005, NTCIR.

[6]  Ellen M. Voorhees,et al.  Overview of the TREC 2002 Question Answering Track , 2003, TREC.

[7]  Sanda M. Harabagiu,et al.  Experiments with Interactive Question Answering in Complex Scenarios , 2004, Workshop On Pragmatics Of Question Answering.

[8]  Tomek Strzalkowski,et al.  HITIQA: A Data Driven Approach to Interactive Question Answering: A Preliminary Report , 2003, New Directions in Question Answering.

[9]  Masaki Murata,et al.  Japanese Question-Answering System Using Decreased Adding with Multiple Answers , 2004, NTCIR.

[10]  Noriko Kando,et al.  Are open-domain question answering technologies useful for information access dialogues?---an empirical study and a proposal of a novel challenge , 2005, TALIP.

[11]  Kenji Ishida,et al.  Three Systems and One Verifier - HOKUM's Participation in QAC3 of NTCIR-5 , 2005, NTCIR.

[12]  Tsuneaki Kato,et al.  Question Answering Challenge for Information Access Dialogue - Overview of NTCIR4 QAC2 Subtask , 2004, NTCIR.

[13]  Takashi Yukawa,et al.  Synthesis of Multiple Answer Evaluation Measures using a Machine Learning Technique for a QA System , 2005, NTCIR.

[14]  福本 淳一,et al.  Question answering challenge (QAC1) , 2002 .