KILM: Knowledge Injection into Encoder-Decoder Language Models

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5013–5035 July 9-14, 2023 ©2023 Association for Computational Linguistics KILM: Knowledge Injection into Encoder-Decoder Language Models Yan Xu1,2∗, Mahdi Namazifar1, Devamanyu Hazarika1, Aishwarya Padmakumar1, Yang Liu1, Dilek Hakkani-Tür1 1Amazon Alexa AI 2Hong Kong University of Science and Technology yxucb@connect.ust.hk, mahdinam@amazon.com, dvhaz@amazon.com padmakua@amazon.com, yangliud@amazon.com, hakkanit@amazon.com Abstract Large pre-trained language models (PLMs) have been shown to retain implicit knowledge within their parameters. To enhance this im- plicit knowledge, we propose Knowledge In- jection into Language Models (KILM), a novel approach that injects entity-related knowledge into encoder-decoder PLMs, via a generative knowledge infilling objective through contin- ued pre-training. This is done without architec- tural modifications to the PLMs or adding ad- ditional parameters. Experimental results over a suite of knowledge-intensive tasks spanning numerous datasets show that KILM enables models to retain more knowledge and halluci- nate less while preserving their original perfor- mance on general NLU and NLG tasks. KILM also demonstrates improved zero-shot perfor- mances on tasks such as entity disambiguation, outperforming state-of-the-art models having 30x more parameters.1 1 Introduction Large pre-trained language models (PLMs) (Rad- ford et al., 2019; Lewis et al., 2020a; Raffel et al., 2020) have achieved great success across all NLP tasks. However, recent studies also reveal that PLMs are susceptible to memorizing the pre- training corpora rather than capturing the knowl- edge within them (Niven and Kao, 2019; Talmor et al., 2020; Yasunaga et al., 2022; Li et al., 2022). Particularly for generation tasks, PLMs are notori- ous for hallucinating text that is factually incorrect or hard to verify (Logan et al., 2019; Sun et al., 2020; Lin et al., 2020; Longpre et al., 2021). To address these issues, one approach is to retrieve relevant knowledge and integrate it explicitly with PLMs (He et al., 2020; Liu et al., 2021b). Another direction is incorporating the additional knowledge sources into the pre-training step (Zhang et al., ∗Work done in part while Yan was an intern at Amazon Alexa AI. 1The code is available at https://github.com/alexa/kilm. 2019; Xiong et al., 2019; Liu et al., 2022; Wang et al., 2021b). While the former suffers from the is- sue of falling back on the models themselves with- out retrieved information (Krishna et al., 2021), knowledge-focused pre-training can be comple- mentary to those methods (Longpre et al., 2021) and shows its advantage on generalization. In this paper, we propose an approach for inject- ing knowledge into encoder-decoder PLMs, such as BART, as a continued pre-training process. We refer to it as Knowledge Injection into Language Models (KILM). Instead of introducing additional parameters to PLMs or modifying the model ar- chitectures to incorporate additional knowledge, KILM infills knowledge sentences by adopting a novel knowledge infilling objective that includes a knowledge reconstruction step in addition to the original pre-training objectives of BART. The aim of KILM is to teach PLMs additional content about concepts and entities that they en- counter in a given context, so that the …

Content truncated to 500 words