WikiQA: A Challenge Dataset for Open-Domain Question Answering

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2013–2018, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. WIKIQA: A Challenge Dataset for Open-Domain Question Answering Yi Yang∗ Georgia Institute of Technology Atlanta, GA 30308, USA yiyang@gatech.edu Wen-tau Yih Christopher Meek Microsoft Research Redmond, WA 98052, USA {scottyih,meek}@microsoft.com Abstract We describe the WIKIQA dataset, a new publicly available set of question and sen- tence pairs, collected and annotated for re- search on open-domain question answer- ing. Most previous work on answer sen- tence selection focuses on a dataset cre- ated using the TREC-QA data, which includes editor-generated questions and candidate answer sentences selected by matching content words in the question. WIKIQA is constructed using a more nat- ural process and is more than an order of magnitude larger than the previous dataset. In addition, the WIKIQA dataset also in- cludes questions for which there are no correct sentences, enabling researchers to work on answer triggering, a critical com- ponent in any QA system. We compare several systems on the task of answer sen- tence selection on both datasets and also describe the performance of a system on the problem of answer triggering using the WIKIQA dataset. 1 Introduction Answer sentence selection is a crucial subtask of the open-domain question answering (QA) prob- lem, with the goal of extracting answers from a set of pre-selected sentences (Heilman and Smith, 2010; Yao et al., 2013; Severyn and Moschitti, 2013). In order to conduct research on this im- portant problem, Wang et al. (2007) created a dataset, which we refer to by QASENT, based on the TREC-QA data. The QASENT dataset chose questions in TREC 8-13 QA tracks and selected sentences that share one or more non-stopwords from the questions. Although QASENT has since ∗Work conducted while interning at Microsoft Research. become the benchmark dataset for the answer se- lection problem, its creation process actually in- troduces a strong bias in the types of answers that are included. The following example illustrates an answer that does not share any content words with the question and would not be selected: Q: How did Seminole war end? A: Ultimately, the Spanish Crown ceded the colony to United States rule. One significant concern with this approach is that the lexical overlap will make sentence selection easier for the QASENT dataset and might inflate the performance of existing systems in more natu- ral settings. For instance, Yih et al. (2013) find that simple word matching methods outperform many sophisticated approaches on the dataset. We ex- plore this possibility in Section 3. A second, more subtle challenge for question answering is that it normally assumes that there is at least one correct answer for each question in the candidate sentences. During the data construction procedures, all the questions without correct an- swers are manually discarded.1 We address a new challenge of answer triggering, an important com- ponent in QA systems, where the goal is to detect whether there exist correct answers in …

Content truncated to 500 words