AmbigQA: Answering Ambiguous Open-domain Questions

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5783–5797, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5783 AMBIGQA: Answering Ambiguous Open-domain Questions Sewon Min1,2 , Julian Michael1, Hannaneh Hajishirzi1,3 , Luke Zettlemoyer1,2 1University of Washington 2Facebook AI Research 3Allen Institute for Artificial Intelligence {sewon,julianjm,hannaneh,lsz}@cs.washington.edu Abstract Ambiguity is inherent to open-domain ques- tion answering; especially when exploring new topics, it can be difficult to ask questions that have a single, unambiguous answer. In this paper, we introduce AMBIGQA, a new open-domain question answering task which involves finding every plausible answer, and then rewriting the question for each one to re- solve the ambiguity. To study this task, we con- struct AMBIGNQ, a dataset covering 14,042 questions from NQ-OPEN, an existing open- domain QA benchmark. We find that over half of the questions in NQ-OPEN are ambigu- ous, with diverse sources of ambiguity such as event and entity references. We also present strong baseline models for AMBIGQA which we show benefit from weakly supervised learn- ing that incorporates NQ-OPEN, strongly sug- gesting our new task and data will support sig- nificant future research effort. Our data and baselines are available at https://nlp.cs. washington.edu/ambigqa. 1 Introduction In the open-domain setting, it can be difficult to formulate clear and unambiguous questions. For example, Figure 1 shows a Google search query (Kwiatkowski et al., 2019) that, perhaps sur- prisingly, has two possible interpretations given the evidence in Wikipedia. Although open-domain question answering (QA) systems aim to answer any factoid question (Voorhees et al., 1999), exist- ing methods assume questions have a single well- defined answer. Nonetheless, ambiguity arises fre- quently in open-domain QA, where questions are written during information gathering (e.g., search queries) without knowledge of the answer. As we will see in Section 4, over 50% of the questions we sampled from a set of Google search queries are ambiguous. Furthermore, identifying ambigui- ties is difficult both for humans and machines. As Figure 1: An AMBIGNQ example where the prompt question (top) appears to have a single clear answer, but is actually ambiguous upon reading Wikipedia. AM- BIGQA requires producing the full set of acceptable answers while differentiating them from each other us- ing disambiguated rewrites of the question. shown in Figure 1, ambiguity is a function of both the question and the evidence provided by a large text corpus. To study this challenge, we introduce AM- BIGQA (Answering Ambiguous Open-domain Questions), a new task which involves disambiguat- ing and answering potentially ambiguous questions. Specifically, the model must (1) find a set of dis- tinct, equally plausible answers to the question, and (2) provide minimal yet unambiguous rewrites of the question that clarify the interpretation which leads to each answer. Figure 1 shows two such disambiguated questions and their answers. To support the study of this task, we construct a dataset called AMBIGNQ using 14,042 questions from an open-domain version of NATURAL QUES- TIONS (Kwiatkowski et al., 2019), denoted NQ- OPEN. For each question, annotators search for, 5784 …

Content truncated to 500 words