Probabilistic Assumptions Matter: Improved Models for Distantly-Supervised Document-Level Question Answering

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5657–5667 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 5657 Probabilistic Assumptions Matter: Improved Models for Distantly-Supervised Document-Level Question Answering Hao Cheng∗, Ming-Wei Chang†, Kenton Lee†, Kristina Toutanova† ∗Microsoft Research chehao@microsoft.com †Google Research {mingweichang, kentonl, kristout}@google.com Abstract We address the problem of extractive question answering using document-level distant super- vision, pairing questions and relevant docu- ments with answer strings. We compare previ- ously used probability space and distant super- vision assumptions (assumptions on the corre- spondence between the weak answer string la- bels and possible answer mention spans). We show that these assumptions interact, and that different configurations provide complemen- tary benefits. We demonstrate that a multi- objective model can efficiently combine the advantages of multiple assumptions and out- perform the best individual formulation. Our approach outperforms previous state-of-the-art models by 4.3 points in F1 on TriviaQA-Wiki and 1.7 points in Rouge-L on NarrativeQA summaries.1 1 Introduction Distant supervision assumptions have enabled the creation of large-scale datasets that can be used to train fine-grained extractive short answer ques- tion answering (QA) systems. TriviaQA (Joshi et al., 2017).One example is There the au- thors utilized a pre-existing set of Trivia question- answer string pairs and coupled them with rele- vant documents, such that, with high likelihood, the documents support answering the questions (see Fig. 1 for an illustration). Another example is the NarrativeQA dataset (Koˇcisk´y et al., 2018), where crowd-sourced abstractive answer strings were used to weakly supervise answer mentions in the text of movie scripts or their summaries. In this work, we focus on the setting of document- level extractive QA, where distant supervision is specified as a set A of answer strings for an input question-document pair. 1Based on the TriviaQA-Wiki leaderboard, our approach was the SOTA when this work was submitted on Dec 04, 2019. Question: How is Joan Molinsky better known? Answer: Joan Rivers : { Joan Rivers, Diary of a Mad Diva } P1: Joan Alexandra Molinsky, known professionally as Joan Rivers, was an American comedian, actress, writer, producer, and television host. … Joan Rivers was strongly influenced by Lenny Bruce. … P2: … She received a Grammy Award for Best Spoken Word Album for her book, Diary of a Mad Diva. … P3: Joan Alexandra Molinsky was born on June 8, 1933, in Brooklyn, New York. … Before entering show business, she chose Joan Rivers as her stage name. … Question: Where do the dancers purify themselves? Answer: in the spring at mount helicon mount helicon : { in the spring at mount helicon, mount helicon } P1: The play begins with three pages … P2: The courtiers … She sentences them to make reparation and to purify themselves by bathing in the spring at mount helicon. The figure of Actaeon in the play may represent ... TriviaQA NarrativeQA Figure 1: TriviaQA and NarrativeQA examples. In the Triv- iaQA example, there are three occurrences of the original an- swer string …

Content truncated to 500 words