Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2226 - 2243 July 10-15, 2022 ©2022 Association for Computational Linguistics Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks Akari Asai∗ University of Washington Matt Gardner∗ Microsoft Semantic Machines {akari,hannaneh}@cs.washington.edu mattgardner@microsoft.com Hannaneh Hajishirzi University of Washington Allen Institute for AI Abstract Retrieval-augmented generation models have shown state-of-the-art performance across many knowledge-intensive NLP tasks such as open-domain question answering and fact ver- ification. These models are trained to gener- ate a final output given retrieved passages that can be irrelevant to an input query, leading to learning spurious cues or memorization. This work introduces a method to incorporate evi- dentiality of passages—whether a passage con- tains correct evidence to support the output— into training the generator. We introduce a multi-task learning framework to jointly gen- erate the final output and predict the eviden- tiality of each passage. Furthermore, we in- troduce a new task-agnostic method for obtain- ing high-quality silver evidentiality labels, ad- dressing the issues of gold evidentiality labels being unavailable in most domains. Our exper- iments on five datasets across three knowledge- intensive tasks show that our new evidentiality- guided generator significantly outperforms its direct counterpart on all of them, and advances the state of the art on three of them. Our anal- ysis shows that the multi-task learning and sil- ver evidentiality mining play key roles.1 1 Introduction Knowledge-intensive tasks, including open-domain Question Answering (QA) and fact verification, re- quire evidence passages related to an input query to be retrieved from a large collection of passages (e.g., Wikipedia). Recently, most successful meth- ods use retrieval-augmented generation (Lewis et al., 2020c; Izacard and Grave, 2021b), which is a pipeline approach of first training a retriever model (Karpukhin et al., 2020) for retrieving pas- sages and then independently training a generator model (Lewis et al., 2020a; Raffel et al., 2020) given the passages. ∗Work done in part while at the Allen Institute for AI 1Our code is available at https://github.com/ AkariAsai/evidentiality_qa. question 1: The Fate of the Furious release date in india? The Fate of the Furious The Fate of the Furious was released in the United States on April 14, 2017. The Fate of the Furious The release across major markets such as Australia, the United Kingdom, China, and India, beginning on April 12, 2017. answer: April 12, 2017 | prediction: April 14, 2017 claim: The first ‘fast and furious’ film was filmed in 2001. gold class: REFUTE | prediction: SUPPORT The Fast and the Furious (2001 film) principal photography began in July 2000 and lasted until that October. The Fast and the Furious (2001 film) Fast & Furious is a media franchise, and the first film was released in 2001 Open Question Answering (input x: question, output y: answer) Fact verification (input x: claim, output y: classification label ) question 2: How many countries India share the border with? India India is seventh largest country. Borders of India India shares …

Content truncated to 500 words