Overview of the NTCIR-12 QA Lab-2 Task

The NTCIR-12 QA Lab-2 task aims at the real-world complex Question Answering (QA) technologies using Japanese university entrance exams and their English translation on the subject of “World history”. The exam questions are roughly divided into multiple-choice and free-description styles, and have various question formats, which are essay, factoid, slot-filling, true-or-false and so on. We conducted three phases of formal runs, and collaborated on Phase-2 Japanese subtask with the Todai Robot Project. Twelve teams submitted 148 runs in total. We describe the used data, the hierarchy of question formats, formal run results, and comparison between human marks and automatic evaluation scores for essay questions. Categories and Subject Descriptions H.3.4 [INFORMATION STORAGE AND RETRIEVAL]: Systems and Software Performance evaluation (efficiency and effectiveness), Question-answering (fact retrieval) systems.