Read before Generate! Faithful Long Form Question Answering with Machine Reading

Findings of the Association for Computational Linguistics: ACL 2022, pages 744 - 756 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Read before Generate! Faithful Long Form Question Answering with Machine Reading Dan Su1∗, Xiaoguang Li2, Jindi Zhang3, Lifeng Shang2, Xin Jiang2, Qun Liu2, and Pascale Fung1 1Center for Artificial Intelligence Research (CAiRE) The Hong Kong University of Science and Technology, Clear Water Bay, Hong Kong 2Huawei Noah’s Ark Lab 3City University of Hong Kong dsu@connect.ust.hk, lixiaoguang11@huawei.com Abstract Long-form question answering (LFQA) aims to generate a paragraph-length answer for a given question. While current work on LFQA using large pre-trained model for generation are effective at producing fluent and somewhat relevant content, one primary challenge lies in how to generate a faithful answer that has less hallucinated content. We propose a new end-to-end framework that jointly models an- swer generation and machine reading. The key idea is to augment the generation model with fine-grained, answer-related salient infor- mation which can be viewed as an emphasis on faithful facts. State-of-the-art results on two LFQA datasets, ELI5 and MS MARCO, demonstrate the effectiveness of our method, in comparison with strong baselines on automatic and human evaluation metrics. A detailed anal- ysis further proves the competency of our meth- ods in generating fluent, relevant, and more faithful answers. 1 Introduction Long-form question answering (LFQA) is a task to generate an in-depth, paragraph-length answer for a given question (Fan et al., 2019). It is important since many of the everyday questions that humans deal with and pose to search engines require multi- sentence explanations (Khashabi et al., 2021) (e.g. why/how..?). It can be integrated with a search en- gine (Metzler et al., 2021), or a virtual conversation agent, and can also be used to generate explana- tions as a complement to short-phrase answers for open-domain questions (Kwiatkowski et al., 2019; Yang et al., 2018), or to answer open-ended ques- tions like those from Reddit forum “Explain Like I’m Five” (Fan et al., 2019). LFQA is quite a challenging task. It often in- volves searching a large external knowledge source ∗ Work done during an internship at Huawei Noah’s Ark lab Figure 1: An example from MS MARCO (Nguyen et al., 2016) dataset. We highlight the unfaithful snippets from other model. Our model(RBG) generate more factually accurate answer. that contains millions of documents for relevant in- formation. Then it generates a paragraph-length an- swer from those retrieved sources. While the great success in retrieval technique (Guu et al., 2020; Karpukhin et al., 2020; Lee et al., 2019) can be carried over to the LFQA setting, more challenges lie in the generation. First, multiple documents that contain hundreds of tokens need to be considered for generation, raising difficulties in the direct use of current pre-trained language models. Second, as different documents may contain redundant, com- plementary, or contradictory information, how to synthesis the information and generate a faithful answer that has less hallucinated content is even more challenging. While recent work on LFQA (Krishna et al., …

Content truncated to 500 words