Question Answering Challenge (QAC-1): An Evaluation of Question Answering Task at NTCIRWorkshop 3

In this paper we describea questionanswering task, called QuestionAnsweringChallenge (QAC), and its first evaluation (QAC1). Thiswascarried out as a taskof NTCIRWorkshop3 in October2002. In QAC,weaimedto encouragethedevelopmentof practical QA systemsin a general domain and focuson resear ch of user interaction and information extraction. Developmentsof evaluation methodof question answeringsystemandinformationresourcesfor evaluation were alsopurposeof QAC. We havedefined threekindsof taskin QAC, which requirefivepossibleanswer s(Task1),onlyoneanswer (Task2) andoneanswerto relatedquestion(Task3). Wehaveprepared200questions for Task1 andTask2 and40 questions for Task3 at Formal Runandabout 900questions for additional run. We haveconducted Dry RunandFormal Runevaluation andfinally there aresixteenparticipants (twoof themare fromtaskorganizer) at QAC1.