Pre-training Language Models with Deterministic Factual Knowledge

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11118 - 11131 December 7-11, 2022 ©2022 Association for Computational Linguistics Pre-training Language Models with Deterministic Factual Knowledge Shaobo Li1, Xiaoguang Li2, Lifeng Shang2, Chengjie Sun1, Bingquan Liu1, Zhenzhou Ji1, Xin Jiang2 and Qun Liu2 1Harbin Institute of Technology, 2Huawei Noah’s Ark Lab shli@insun.hit.edu.cn, {sunchengjie, liubq, jizhenzhou}@hit.edu.cn {lixiaoguang11, shang.lifeng, Jiang.Xin, qun.liu}@huawei.com Abstract Previous works show that Pre-trained Language Models (PLMs) can capture factual knowledge. However, some analyses reveal that PLMs fail to perform it robustly, e.g., being sensitive to the changes of prompts when extracting factual knowledge. To mitigate this issue, we propose to let PLMs learn the deterministic relationship between the remaining context and the masked content. The deterministic relationship ensures that the masked factual content can be determin- istically inferable based on the existing clues in the context. That would provide more stable patterns for PLMs to capture factual knowledge than randomly masking. Two pre-training tasks are further introduced to motivate PLMs to rely on the deterministic relationship when filling masks. Specifically, we use an external Knowl- edge Base (KB) to identify deterministic rela- tionships and continuously pre-train PLMs with the proposed methods. The factual knowledge probing experiments indicate that the continu- ously pre-trained PLMs achieve better robust- ness in factual knowledge capturing. Further experiments on question-answering datasets show that trying to learn a deterministic rela- tionship with the proposed methods can also help other knowledge-intensive tasks. 1 Introduction Petroni et al. (2019); Jiang et al. (2020); Shin et al. (2020); Zhong et al. (2021) show that we can successfully extract factual knowledge from Pre- trained Language Models (PLMs) using cloze-style prompts such as “The director of the film Saving Private Ryan is [MASK].” Some recent works (Cao et al., 2021; Pörner et al., 2020) find that the PLMs may rely on superficial cues to achieve that and can not respond robustly. Table 1 gives examples of inconsistent predictions exposed by changing the surface forms of prompts on the same fact. This phenomenon questions whether PLMs can robustly capture factual knowledge through Masked Language Modeling (MLM) (Devlin et al., Cloze-style Prompt and Prediction Is Correct? War Horse is an American war film di- rected by Steven Spielberg. ✓ The director of the American war film War Horse is Keanu Reeves. ✗ Christopher Nolan is the director of the American war film War Horse. ✗ Table 1: A PLM could gives inconsistent results when probing the same fact with different prompts. The un- derlined words are the predictions. 2018) and further intensify us to inspect the masked contents in the pre-training samples. After review- ing several masking methods, we find that they fo- cus on limiting the granularity of masked contents, e.g., restricting the masked content to be entities and then randomly masking the entities (Guu et al., 2020), and pay less attention to checking whether the obtained MLM samples are appropriate for fac- tual knowledge capturing. For instance, when we want PLMs to capture the …

Content truncated to 500 words