Can Generative Pre-trained Language Models Serve As Knowledge Bases for Closed-book QA?

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3241–3251 August 1–6, 2021. ©2021 Association for Computational Linguistics 3241 Can Generative Pre-trained Language Models Serve as Knowledge Bases for Closed-book QA? Cunxiang Wang♠♣∗, Pai Liu♣∗and Yue Zhang♣♥† ♠Zhejiang University, China ♣School of Engineering, Westlake University, China ♥Institute of Advanced Technology, Westlake Institute for Advanced Study, China {wangcunxiang, zhangyue, liupai}@westlake.edu.cn Abstract Recent work has investigated the interesting question using pre-trained language models (PLMs) as knowledge bases for answering open questions. However, existing work is limited in using small benchmarks with high test-train overlaps. We construct a new dataset of closed-book QA using SQuAD, and in- vestigate the performance of BART. Experi- ments show that it is challenging for BART to remember training facts in high precision, and also challenging to answer closed-book questions even if relevant knowledge is re- tained. Some promising directions are found, including decoupling the knowledge memoriz- ing process and the QA finetune process, forc- ing the model to recall relevant knowledge when question answering. 1 Introduction Large-scare pre-trained language models (PLMs) such as BERT (Devlin et al., 2019), GPT (Radford et al., 2018) have significantly improved the perfor- mance of NLP tasks (Radford et al., 2019). There is increasing evidence showing that PLMs contain world knowledge (Petroni et al., 2019; Zhou et al., 2020; Talmor et al., 2020). As a result, recent re- search considers generative PLMs such as T5 (Raf- fel et al., 2020) and BART (Lewis et al., 2020a) for Closed-book QA, which has only question-answer pairs without external knowledge source. For ex- ample, after being finetuned on a few QA pairs, a generative LM can directly output “Florence” af- ter being given the question “Where was Dante born?”. Roberts et al. (2020) find that generative PLMs can store and use knowledge as they can achieve relatively high performance in closed-book QA task on three datasets. However, Lewis et al. (2020b) find that the excellent results are mainly ∗Equal contribution †The corresponding author Figure 1: Process of generative PLMs for closed-book QA. (1) BART performs poorly on closed-book QA after QA finetuning; (2) We LM-finetune BART with related passages to feed knowledge and use a recit- ing task to evaluate how much knowledge the LM- finetuned model memorizes; (3) Though memorizing most needed knowledge, BART still faces challenge on closed-book QA after QA finetuning. due to high question/answer overlap rates between training and testing data. Existing research leaves many open questions on the potential of generative pre-trained LMs on closed-book QA. For example, the used datasets consist of question-answer pairs only, and there is no mechanism to control what factual knowledge is already used to train a generative PLM before taking the closed-book questions. In addition, the high overlapping rates between training and testing questions and answers make it difficult to under- stand whether the answer that a model gives comes from its inherent knowledge or superficial cues in training data. …

Content truncated to 500 words