TempoWiC: An Evaluation Benchmark for Detecting Meaning Shift in Social Media

Proceedings of the 29th International Conference on Computational Linguistics, pages 3353–3359 October 12–17, 2022. 3353 TempoWiC: An Evaluation Benchmark for Detecting Meaning Shift in Social Media Daniel Loureiro♢ Aminette D’Souza∗♢, Areej Nasser Muhajab∗♢, Isabella A. White∗♢, Gabriel Wong∗♢ Luis Espinosa Anke♢♡, Leonardo Neves♣, Francesco Barbieri♣, Jose Camacho-Collados♢ ♢ Cardiff NLP, School of Computer Science and Informatics, Cardiff University, UK ♣ Snap Inc., Santa Monica, California, USA; ♡AMPLYFI, UK ♢boucanovaloureirod@cardiff.ac.uk, cardiffnlp.contact@gmail.com Abstract Language evolves over time, and word mean- ing changes accordingly. This is especially true in social media, since its dynamic nature leads to faster semantic shifts, making it challenging for NLP models to deal with new content and trends. However, the number of datasets and models that specifically address the dynamic nature of these social platforms is scarce. To bridge this gap, we present TempoWiC, a new benchmark especially aimed at accelerating research in social media-based meaning shift. Our results show that TempoWiC is a challeng- ing benchmark, even for recently-released lan- guage models specialized in social media. 1 Introduction One of the most studied challenges in NLP is lexi- cal ambiguity. Solutions include word sense disam- biguation (Navigli, 2009) or entity linking (Ling et al., 2015), where words are linked to sense inven- tories such as WordNet (Miller, 1995) or Wikipedia. Recently, notable progress has been made with the advent of Language Models (LMs) and contextual- ized embeddings, crucially well equipped for mod- eling meaning in context (Pilehvar and Camacho- Collados, 2020; Bevilacqua et al., 2020). One notable limitation with current lexical se- mantics benchmarks, however, is that they are typ- ically clean and time-invariant, where standard grammar is the norm, and have little to no account of language usage in real-world platforms like so- cial media. However, there is ample agreement that modeling changes in language and topic distribu- tions is crucial for modern NLP (Loureiro et al., 2022a). Thus, there is a rich body of literature con- cerned with, e.g., adapting existing word represen- tations (mainly word embeddings) diachronically (Hamilton et al., 2016; Szymanski, 2017; Rosen- feld and Erk, 2018; Hofmann et al., 2021), expos- ing LMs to time-specific data (Lazaridou et al., ∗Annotation team. 2021), or temporal adaptation in general (Luu et al., 2021; Agarwal and Nenkova, 2022; Jin et al., 2021; Loureiro et al., 2022a). The lack of real-world data to serve as ground truth has typically limited the evaluation of diachronic word-level NLP models. This limitation has been addressed in a myriad of ways, e.g., by comparing distributional similari- ties with human judgments (Gulordava and Baroni, 2011), contrasting change vs. frequency (Hamil- ton et al., 2016), comparing time-sensitive repre- sentations with stripped-down versions (Frermann and Lapata, 2016) or, more recently, determining whether a word has acquired new senses over time by looking at relatively large targeted subcorpora (Van Hee et al., 2018), or probing for diachronic awareness in settings reminiscent of knowledge base completion (Dhingra et al., 2021; Hofmann et al., 2021). Despite the above, few works have attempted to model the connection …

Content truncated to 500 words