Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledge

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5469–5485 July 9-14, 2023 ©2023 Association for Computational Linguistics Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledge Yasumasa Onoe, Michael J.Q. Zhang, Shankar Padmanabhan, Greg Durrett, Eunsol Choi Department of Computer Science The University of Texas at Austin yasumasa@utexas.edu Abstract Pre-trained language models (LMs) are used for knowledge intensive tasks like question an- swering, but their knowledge gets continuously outdated as the world changes. Prior work has studied targeted updates to LMs, injecting indi- vidual facts and evaluating whether the model learns these facts while not changing predic- tions on other contexts. We take a step forward and study LMs’ abilities to make inferences based on injected facts (or propagate those facts): for example, after learning that some- thing is a TV show, does an LM predict that you can watch it? We study this with two cloze- style tasks: an existing dataset of real-world sentences about novel entities (ECBD) as well as a new controlled benchmark with manually designed templates requiring varying levels of inference about injected knowledge. Surpris- ingly, we find that existing methods for updat- ing knowledge (gradient-based fine-tuning and modifications of this approach) show little prop- agation of injected knowledge. These methods improve performance on cloze instances only when there is lexical overlap between injected facts and target inferences. Yet, prepending entity definitions in an LM’s context improves performance across all settings, suggesting that there is substantial headroom for parameter- updating approaches for knowledge injection. 1 Introduction Pre-trained language models (LMs) acquire com- prehensive real-world knowledge from massive amounts of pre-training data, allowing them to use this knowledge effectively in downstream tasks. However, without continual updating, the knowl- edge contained within these backend LMs will eventually become outdated. This temporal mis- match affects model performance on downstream tasks (Zhang and Choi, 2021; Dhingra et al., 2022a; Lazaridou et al., 2021; Jang et al., 2022b). As LMs become more widely deployed, their knowledge Knowledge Editing (Prior work) Entity Knowledge Propagation (This work) Update: : Who is the Prime Minister of the UK? ; Evaluation (Updated fact): : Who is the UK’s PM? Update: Evaluation (Inference based on the updated fact): : The English Game is a British historical sports drama television miniseries about the origins of modern association football in England. : The English Game is all about a story of [MASK] people. Liz Truss Rishi Sunak Liz Truss Rishi Sunak Update( , , ) : Rishi Sunak Rishi Sunak funny athletic unlawful Update( , ) Figure 1: Knowledge editing tasks. We study a challeng- ing entity knowledge propagation task where language models should make inferences after learning entities from their definitions. This differs from past knowledge editing which evaluates paraphrases of injected facts. should be synced with the current state of the world while maintaining reasonable deployment costs. Prior work has investigated knowledge editing in pre-trained LMs, updating model parameters to alter outputs to match …

Content truncated to 500 words