Unsupervised Improvement of Factual Knowledge in Language Models

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2960–2969 May 2-6, 2023 ©2023 Association for Computational Linguistics Unsupervised Improvement of Factual Knowledge in Language Models Nafis Sadeq∗, Byungkyu Kang†, Prarit Lamba†, Julian McAuley∗ Intuit† and UC San Diego∗ {nsadeq,jmcauley}@ucsd.edu {Jay_Kang,Prarit_Lamba}@intuit.com Abstract Masked language modeling (MLM) plays a key role in pretraining large language models. But the MLM objective is often dominated by high-frequency words that are sub-optimal for learning factual knowledge. In this work, we propose an approach for influencing MLM pre- training in a way that can improve language model performance on a variety of knowledge- intensive tasks. We force the language model to prioritize informative words in a fully unsuper- vised way. Experiments demonstrate that the proposed approach can significantly improve the performance of pretrained language models on tasks such as factual recall, question answer- ing, sentiment analysis, and natural language inference in a closed-book setting. 1 Introduction Pretrained language models (PLMs) such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), BART (Lewis et al., 2020), T5 (Raffel et al., 2020) use a Masked Language Modeling (MLM) objective during pretraining. However, a traditional MLM objective may not be optimal for knowledge- intensive tasks (Peters et al., 2019). It has been shown that language models can benefit from incor- porating knowledge within the training objective in the form of entity embeddings (Peters et al., 2019; Zhang et al., 2019), knowledge retriever (Guu et al., 2020), knowledge embedding (Wang et al., 2021; Sun et al., 2020) or augmented pretraining cor- pora created from Knowledge Graphs (Agarwal et al., 2021). Despite their effectiveness, these approaches rely on existing knowledge bases and entity embeddings to incorporate knowledge within the training objective. These resources are expen- sive to construct and may not be available for all languages and domains (Huang et al., 2022). In this work, we propose a pretraining approach that can achieve better performance on knowledge- intensive tasks without using any existing knowl- edge base. We combine two key strategies to in- fluence MLM objective. Firstly, the tokens with higher informative relevance should be masked more frequently (Sadeq et al., 2022). Secondly, mistakes on informative tokens should be penal- ized more severely. The informative relevance of the tokens can be computed efficiently with a one- pass computation on the pretraining corpora. Ex- periments demonstrate that the proposed training strategy can help the language model achieve bet- ter performance on the factual knowledge recall benchmark LAMA (Petroni et al., 2019), extractive question answering (QA) benchmark SQuAD (Ra- jpurkar et al., 2016, 2018), prompt based sentiment analysis and natural language inference (NLI) tasks in AutoPrompt (Shin et al., 2020). The key contribution of this work is proposing a completely unsupervised stand-alone MLM pre- training objective for language models that can significantly improve performance on knowledge- intensive tasks. Unlike prior works in the area, our method does not require existing knowledge bases to incorporate knowledge during pretraining. We make the code publicly available. …

Content truncated to 500 words