Collecting a Why-Question Corpus for Development and Evaluation of an Automatic QA-System

Proceedings of ACL-08: HLT, pages 443–451, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics Collecting a Why-question corpus for development and evaluation of an automatic QA-system Joanna Mrozinski Edward Whittaker Department of Computer Science Tokyo Institute of Technology 2-12-1-W8-77 Ookayama, Meguro-ku Tokyo 152-8552 Japan {mrozinsk,edw,furui}@furui.cs.titech.ac.jp Sadaoki Furui Abstract Question answering research has only recently started to spread from short factoid questions to more complex ones. One significant chal- lenge is the evaluation: manual evaluation is a difficult, time-consuming process and not ap- plicable within efficient development of sys- tems. Automatic evaluation requires a cor- pus of questions and answers, a definition of what is a correct answer, and a way to com- pare the correct answers to automatic answers produced by a system. For this purpose we present a Wikipedia-based corpus of Why- questions and corresponding answers and arti- cles. The corpus was built by a novel method: paid participants were contacted through a Web-interface, a procedure which allowed dy- namic, fast and inexpensive development of data collection methods. Each question in the corpus has several corresponding, partly over- lapping answers, which is an asset when es- timating the correctness of answers. In ad- dition, the corpus contains information re- lated to the corpus collection process. We be- lieve this additional information can be used to post-process the data, and to develop an auto- matic approval system for further data collec- tion projects conducted in a similar manner. 1 Introduction Automatic question answering (QA) is an alternative to traditional word-based search engines. Instead of returning a long list of documents more or less re- lated to the query parameters, the aim of a QA sys- tem is to isolate the exact answer as accurately as possible, and to provide the user only a short text clip containing the required information. One of the major development challenges is eval- uation. The conferences such as TREC1, CLEF2 and NTCIR3 have provided valuable QA evaluation methods, and in addition produced and distributed corpora of questions, answers and corresponding documents. However, these conferences have fo- cused mainly on fact-based questions with short an- swers, so called factoid questions. Recently more complex tasks such as list, definition and discourse- based questions have also been included in TREC in a limited fashion (Dang et al., 2007). More complex how- and why-questions (for Asian languages) were also included in the NTCIR07, but the provided data comprised only 100 questions, of which some were also factoids (Fukumoto et al., 2007). Not only is the available non-factoid data quite limited in size, it is also questionable whether the data sets are us- able in development outside the conferences. Lin and Katz (2006) suggest that training data has to be more precise, and, that it should be collected, or at least cleaned, manually. Some corpora of why-questions have been col- lected manually: corpora described in (Verberne et al., 2006) and (Verberne et al., 2007) both com- prise fewer than 400 questions and corresponding answers (one or two per …

Content truncated to 500 words