Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answering

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3932–3945 August 1–6, 2021. ©2021 Association for Computational Linguistics 3932 Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answering Najoung Kim†,∗, Ellie Pavlickφ,δ, Burcu Karagol Ayanδ, Deepak Ramachandranδ,∗ †Johns Hopkins University φBrown University δGoogle Research n.kim@jhu.edu {epavlick,burcuka,ramachandrand}@google.com Abstract Many Question-Answering (QA) datasets contain unanswerable questions, but their treatment in QA systems remains primi- tive. Our analysis of the Natural Ques- tions (Kwiatkowski et al., 2019) dataset re- veals that a substantial portion of unanswer- able questions (∼21%) can be explained based on the presence of unverifiable presupposi- tions. Through a user preference study, we demonstrate that the oracle behavior of our proposed system—which provides responses based on presupposition failure—is preferred over the oracle behavior of existing QA sys- tems. Then, we present a novel framework for implementing such a system in three steps: presupposition generation, presupposition ver- ification, and explanation generation, report- ing progress on each. Finally, we show that a simple modification of adding presuppositions and their verifiability to the input of a com- petitive end-to-end QA system yields modest gains in QA performance and unanswerabil- ity detection, demonstrating the promise of our approach. 1 Introduction Many Question-Answering (QA) datasets includ- ing Natural Questions (NQ) (Kwiatkowski et al., 2019) and SQuAD 2.0 (Rajpurkar et al., 2018) con- tain questions that are unanswerable. While unan- swerable questions constitute a large part of exist- ing QA datasets (e.g., 51% of NQ, 36% of SQuAD 2.0), their treatment remains primitive. That is, (closed-book) QA systems label these questions as Unanswerable without detailing why, as in (1): (1) a. Answerable Q: Who is the current monarch of the UK? System: Elizabeth II. ∗Corresponding authors, †Work done at Google b. Unanswerable Q: Who is the current monarch of France? System: Unanswerable. Unanswerability in QA arises due to a multitude of reasons including retrieval failure and malformed questions (Kwiatkowski et al., 2019). We focus on a subset of unanswerable questions—namely, questions containing failed presuppositions (back- ground assumptions that need to be satisfied). Questions containing failed presuppositions do not receive satisfactory treatment in current QA. Under a setup that allows for Unanswerable as an answer (as in several closed-book QA systems; Fig- ure 1, left), the best case scenario is that the system correctly identifies that a question is unanswerable and gives a generic, unsatisfactory response as in (1-b). Under a setup that does not allow for Unan- swerable (e.g., open-domain QA), a system’s at- tempt to answer these questions results in an inaccu- rate accommodation of false presuppositions. For example, Google answers the question Which lin- guist invented the lightbulb? with Thomas Edison, and Bing answers the question When did Marie Curie discover Uranium? with 1896 (retrieved Jan 2021). These answers are clearly inappropriate, because answering these questions with any name or year endorses the false presuppositions Some lin- guist invented the lightbulb and Marie Curie discov- ered Uranium. Failures of this …

Content truncated to 500 words