Time-Aware Language Models as Temporal Knowledge Bases

Time-Aware Language Models as Temporal Knowledge Bases Bhuwan Dhingra∗∗∗ Jeremy R. Cole∗ Julian Martin Eisenschlos Daniel Gillick Jacob Eisenstein William W. Cohen Google Research {bdhingra,jrcole,eisenjulian,dgillick,jeisenstein,wcohen}@google.com Abstract Many facts come with an expiration date, from the name of the President to the basketball team Lebron James plays for. However, most language models (LMs) are trained on snap- shots of data collected at a specific moment in time. This can limit their utility, especially in the closed-book setting where the pretraining corpus must contain the facts the model should memorize. We introduce a diagnostic dataset aimed at probing LMs for factual knowledge that changes over time and highlight problems with LMs at either end of the spectrum—those trained on specific slices of temporal data, as well asthose trained on a wide range of tempo- ral data. To mitigate these problems, we pro- pose a simple technique for jointly modeling text with its timestamp. This improves mem- orization of seen facts from the training time period, as well as calibration on predictions about unseen facts from future time periods. We also show that models trained with tem- poral context can be efficiently ‘‘refreshed’’ as new data arrives, without the need for re- training from scratch. 1 Introduction Language models (LMs) have been suggested as repositories of real-world knowledge (Petroni et al., 2019) and there is much interest in using them for tasks such as closed-book question an- swering (QA; Roberts et al., 2020), fact verifica- tion (Lee et al., 2020), and dialogue (Adiwardana et al., 2020). Many facts, however, change with time. This raises two questions: Do pretrained LMs learn the appropriate temporal scope for the facts they encode? And what is the best way to update temporally scoped knowledge in pre- trained models? ∗Equal contribution. ∗∗Also affiliated with Duke University, work done at Google. Pretraining corpora for models such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), and GPT (Radford et al., 2019) are typically de- rived from a snapshot of the web crawled at a specific moment in time (Raffel et al., 2020). While the impact on language modeling itself has been highlighted in recent work (e.g., Lazaridou et al., 2021; R¨ottger and Pierrehumbert, 2021; Hombaiah et al., 2021), there are several poten- tial problems specific to the encoding of factual knowledge: • Averaging: For temporally scoped knowl- edge, the model may see conflicting informa- tion, for example, ‘‘Lebron James plays for the Cavaliers / Lakers.’’ Because LM train- ing generally ignores temporal metadata, this can lead to an averaging effect, in which the model has low confidence in any of the correct answers. • Forgetting: Corpora such as Wikipedia and web crawls are constantly growing, with documents distributed non-uniformly across time: There are more recent documents than older ones, both because old documents can be updated and because more web documents are generated recently than in the past. As a result, the model may fail to memorize facts that were valid only during underrepresented periods of time, …

Content truncated to 500 words