Question Answering Challenge (QAC-1): An Evaluation of Question Answering Tasks at the NTCIR Workshop 3

New Directions in Question Answering

Volume

Papers from the 2003 AAAI Spring Symposium

Contents

In this paper we describe the Question Answering Challenge (QAC), a question answering task, and its first evaluation (QAC1). The project was carried out as a task of the NTCIR Workshop 3 in October 2002. One objective of the QAC was to develop practical QA systems in a general domain by focusing on research relating to user interaction and information extraction. Our second objective was to develop an evaluation method for the question answering system and information resources for evaluation. We defined three kinds of tasks in the QAC: Task 1, where questions required five possible answers;, Task 2, where questions had a single answer; and Task 3 ,where there was one answer to a question related to a question in Task 2. We prepared 200 questions for Task 1 and Task 2 and 40 questions for Task 3 at the Formal Run and about 900 questions for the additional run. We conducted a Dry Run and a Formal Run evaluation. There were 16 participants (two of them from among the task organizers) at the QAC1.