WIKITIDE: A Wikipedia-Based Timestamped Definition Pairs Dataset

Proceedings of Recent Advances in Natural Language Processing, pages 207–216 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_023 207 WIKITIDE: A WIKIPEDIA-BASED TIMESTAMPED DEFINITION Pairs Dataset Hsuvas Borkakoty∗, Luis Espinosa-Anke∗⋄ ∗Cardiff NLP, School of Computer Science and Informatics, Cardiff University, UK ⋄AMPLYFI, UK {borkakotyh,espinosaankel}@cardiff.ac.uk Abstract A fundamental challenge in the current NLP context, dominated by language models, comes from the inflexibility of current architectures to “learn” new information. While model-centric solutions like continual learning or parameter- efficient fine-tuning are available, the ques- tion still remains of how to reliably identify changes in language or in the world. In this paper, we propose WikiTiDe, a dataset derived from pairs of timestamped definitions extracted from Wikipedia. We argue that such resource can be helpful for accelerating diachronic NLP, specifically, for training models able to scan knowledge resources for core updates concern- ing a concept, an event, or a named entity. Our proposed end-to-end method is fully auto- matic, and leverages a bootstrapping algorithm for gradually creating a high-quality dataset. Our results suggest that bootstrapping the seed version of WikiTiDe leads to better fine-tuned models. We also leverage fine-tuned models in a number of downstream tasks, showing promising results with respect to competitive baselines1. 1 Introduction Handling new information is one of the most crit- ical (and vastly unresolved) challenges in the cur- rent NLP landscape, mostly because language mod- els (LMs) such as BERT (Devlin et al., 2019), T5 (Raffel et al., 2020), GPT-3 (Brown et al., 2020) or PaLM (Chowdhery et al., 2022) can only learn from information they have seen during pretraining. This is an important limitation when it comes to dealing with updates in the world and language changes alike, since these updates, if not dealt with properly in an LM-centric system, can cause temporal mis- alignment (Luu et al., 2021; Lazaridou et al., 2021; Jang et al., 2022), which is especially harming in 1https://github.com/hsuvas/wiki_ weakly_supervised_classifier-main.git knowledge-intensive tasks, such as closed-book QA. Unsuprisingly, thus, there is a significant body of work concerned with, for instance, updating lan- guage models by pretraining them on in-domain data (Gururangan et al., 2020), editing specific facts (De Cao et al., 2021; Zhu et al., 2020; Dai et al., 2021), continual learning (Agarwal and Nenkova, 2021; Del Tredici et al., 2018; Giulianelli et al., 2020; Dhingra et al., 2022; Loureiro et al., 2022), pre-training with an objective specifically designed to handle infusion of newly coined terms (Yu et al., 2021), or directly modifying the attention mecha- nism to account for temporality (Rosin and Radin- sky, 2022). All these, in addition to the extensive body of work on diachronic and dynamic (contex- tualized and static) word embeddings (Hamilton et al., 2016a; Rudolph et al., 2016; Hamilton et al., 2016b; Rudolph and Blei, 2018; Hofmann et al., 2020). Regardless of the method, however, a critical component of time-aware NLP is to have access to dynamically changing facts about language and the world so that LMs are exposed to them. As Jang et al. (2022) argues, collaborative resources such as …

Content truncated to 500 words